From aeb831e1e048a24a6c2c9ea7761d5131fc3ad2c2 Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Tue, 13 Oct 2026 17:58:24 -0400 Subject: [PATCH 01/90] =?UTF-8?q?=F0=9F=93=9A=20docs(openspec):=20propose?= =?UTF-8?q?=20export,=20import,=20compare=20and=20verify=20for=20prompts?= =?UTF-8?q?=20and=20assistants?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add the OpenSpec change export-import-prompts-and-assistants: a new prompt capability and assistant additions for writing a prompt commit or an assistant to a file, recreating it in another workspace or deployment, comparing a copy with the file, and checking that the copy can run. The task list splits the work into one pull request per group. #807 Export, import, compare, and verify prompts and assistants across LangSmith environments Co-Authored-By: Claude Opus 5.5 --- .../.openspec.yaml | 2 + .../design.md | 145 +++++++++++++++ .../proposal.md | 43 +++++ .../references/langgraph-sdk.md | 13 ++ .../references/langsmith-cli.md | 17 ++ .../references/langsmith-go.md | 13 ++ .../references/langsmith-sdk.md | 13 ++ .../specs/assistant/spec.md | 122 +++++++++++++ .../specs/prompt/spec.md | 166 ++++++++++++++++++ .../tasks.md | 66 +++++++ 10 files changed, 600 insertions(+) create mode 100644 openspec/changes/export-import-prompts-and-assistants/.openspec.yaml create mode 100644 openspec/changes/export-import-prompts-and-assistants/design.md create mode 100644 openspec/changes/export-import-prompts-and-assistants/proposal.md create mode 100644 openspec/changes/export-import-prompts-and-assistants/references/langgraph-sdk.md create mode 100644 openspec/changes/export-import-prompts-and-assistants/references/langsmith-cli.md create mode 100644 openspec/changes/export-import-prompts-and-assistants/references/langsmith-go.md create mode 100644 openspec/changes/export-import-prompts-and-assistants/references/langsmith-sdk.md create mode 100644 openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md create mode 100644 openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md create mode 100644 openspec/changes/export-import-prompts-and-assistants/tasks.md diff --git a/openspec/changes/export-import-prompts-and-assistants/.openspec.yaml b/openspec/changes/export-import-prompts-and-assistants/.openspec.yaml new file mode 100644 index 000000000..5fcc9ecae --- /dev/null +++ b/openspec/changes/export-import-prompts-and-assistants/.openspec.yaml @@ -0,0 +1,2 @@ +schema: spec-driven +created: 2026-10-13 diff --git a/openspec/changes/export-import-prompts-and-assistants/design.md b/openspec/changes/export-import-prompts-and-assistants/design.md new file mode 100644 index 000000000..bc65b0547 --- /dev/null +++ b/openspec/changes/export-import-prompts-and-assistants/design.md @@ -0,0 +1,145 @@ +# Design + +## Context + +`langstar prompt push` and `pull` in `cli/src/commands/prompt.rs` handle one shape of prompt: a single human message, with an optional schema from [#402 structured output prompts](https://github.com/codekiln/langstar/issues/402). `PromptClient` in `sdk/src/prompts.rs` keeps the manifest as raw JSON and has a typed `StructuredPrompt` view that holds messages, the schema and the method. No call asks LangSmith for the model, so neither command sees one. + +`langstar assistant` in `cli/src/commands/assistant.rs` takes `--deployment` on every subcommand and sends requests to that deployment's Agent Server. `AssistantClient` in `sdk/src/assistants.rs` has no `context` field, and its create request cannot set `assistant_id` or `if_exists`. + +Langstar reads one API key and one workspace at a time, from flags, environment variables or the config file. Every error exits with status 1. + +## Goals / Non-Goals + +**Goals:** + +- Carry every field of a prompt commit and an assistant through a file and back without loss. +- Make a repeated import of the same file safe: it finds its earlier copy and changes nothing until the user asks. +- Give users one way to check a copy: the same report shape and exit statuses for both resource types. + +**Non-Goals:** + +- Copy a prompt or assistant between environments in one command. +- Export or import every prompt in a workspace, or every assistant on a deployment, in one run. +- Redesign `prompt create`, `get` and `update`, which [#728 Redesign prompt commands around create/get/update](https://github.com/codekiln/langstar/issues/728) covers. + +## Decisions + +### A file carries each resource between environments + +Export runs with the source environment's credentials and writes a file. Import runs later with the destination's credentials and reads it. Each run uses langstar's existing settings, so a user moving a prompt to another organization sets a different `LANGSMITH_API_KEY` for the import. Langstar needs no second set of credentials and no profiles. The file sits in Git between the two runs, so a reviewer sees the prompt or assistant change in a pull request. + +The LangSmith Go CLI follows the same file workflow, as `references/langsmith-cli.md` shows. Langstar differs from it on purpose. Langstar's export asks for the model and the Go CLI's pull does not. Langstar's import reads the file export writes, while the Go CLI's push wants the bare manifest and rejects its own pull output. Langstar's import also requires an explicit `--workspace-id`, so the destination is named on the command line and never comes from a leftover environment variable. + +### The export file keeps the manifest LangSmith returned + +Export stores the manifest as raw JSON. The typed `StructuredPrompt` struct cannot hold a model binding, placeholders, several message types or partial variables, so routing through it would drop them. Langstar reads the manifest through typed views only to fill in `requires`. + +One prompt export looks like this: + +```json +{ + "kind": "prompt", + "format_version": 1, + "source": { "workspace_id": "", "handle": "my-prompt", "commit_hash": "", "ref": "prod" }, + "repo": { "handle": "my-prompt", "description": "...", "readme": "...", "tags": ["support"], "is_public": false }, + "commit": { "manifest": { "lc": 1, "type": "constructor", "id": ["langchain", "schema", "runnable", "RunnableSequence"], "kwargs": { "first": {}, "last": {} } }, "model_config": {}, "model_provider": "ChatOpenAI", "tags": ["prod"] }, + "requires": { "secrets": ["OPENAI_API_KEY"], "model_provider": "ChatOpenAI" } +} +``` + +An assistant export has `kind: "assistant"`, a `source` with the deployment, assistant ID and version, an `assistant` object with `name`, `description`, `graph_id`, `config`, `context` and `metadata`, and `requires.prompts`. + +The file holds no export time and langstar sorts its keys, so exporting an unchanged resource leaves Git with nothing to commit. Source IDs live only in `source`. + +### Export asks LangSmith for the model + +Export reads the commit with `include=model`, the form the Go SDK uses (`references/langsmith-go.md`). The Python SDK sends the older `include_model=true`, and the LangSmith OpenAPI spec accepts both. With the model included, the manifest becomes a `RunnableSequence` whose last step binds the chat model and its settings, and the response also carries `model_config` and `model_provider` beside the manifest (`references/langsmith-sdk.md`). Creating a commit accepts only `manifest`, `parent_commit` and `description`, so the model reaches the destination inside the manifest. The OpenAPI validation task checks that the destination returns the same model after the push. + +### Import records what it created in a separate file + +Import writes the destination's IDs to `.destinations.json`, keyed by destination workspace ID for prompts and by deployment ID for assistants. The export file stays the same after any number of imports, and a fresh export never wipes out the record of earlier imports. A repeated import reads the record to find the copy it made. For a prompt, the handle already finds the copy, and the record tells langstar whether someone changed the destination since the last import. For an assistant, whose ID differs on every deployment, the record is the only link from the file to its copy. + +### Import changes an existing resource only with --update + +A prompt that exists in the destination gets a new commit only with `--update`, and its parent is the destination's latest commit, so LangSmith keeps the earlier history. An assistant that exists gets updated only with `--update`, which makes the Agent Server save a new version. Without `--update`, import lists the differences and exits with status 2. `--dry-run` reports the action import would take and writes nothing. + +### Compare reports differences by JSON path + +Compare reads the destination, removes the fields the server sets (IDs, version numbers, timestamps, owner and like counts), and walks both JSON documents. Each difference names its JSON Pointer path and both values. Import runs the same comparison after it writes, so a field LangSmith or the Agent Server drops shows up as a difference. + +### Verify checks what the copy needs to run + +Prompt verify looks up each name in `requires.secrets` among the workspace's secrets (`GET /api/v1/workspaces/current/secrets`, which `langstar secrets list` already calls) and looks for a model configuration whose provider matches `requires.model_provider` (`GET /api/v1/playground-settings`, used by `langstar model-config list`). It reports names only. Langstar finds secret names by walking the manifest for LangChain secret references, `{"lc": 1, "type": "secret", "id": ["OPENAI_API_KEY"]}`, the shape the Python SDK's `_manifest_has_secrets` matches. + +Assistant verify checks the graph with `POST /assistants/search` filtered to the graph ID; every graph on a deployment has a default assistant, so an empty result means the deployment does not serve the graph. It checks each prompt in `requires.prompts` with the same lookup `langstar prompt get` uses. + +### The user names an assistant's prompt references + +An assistant refers to a prompt through whatever configuration key its graph reads, so langstar cannot find the references by itself. `assistant export --prompt-ref ` names each one. Prompts should be referred to by handle and tag, such as `support-system:prod`, because a commit hash differs between workspaces. Import carries commit tags across for this reason. + +### Exit statuses + +Status 0 means the resource matches and every check passed. Status 2 means langstar found a difference or a failed check. Status 1 stays the status for every other error, as in every other langstar command. `diff` uses 1 for differences and 2 for errors; langstar keeps 1 for errors so that a script testing for failure sees the same status from every command. + +### Structured output check + +The LangSmith OpenAPI spec has one endpoint that invokes a prompt, `POST /api/v1/prompts/invoke_prompt`. Its body, `InvokePromptPayload`, holds only message templates, `template_format` and `inputs`, with no field for a model or an output schema, and the spec gives no shape for its response. No LangSmith SDK calls it. So the API offers no documented way to show that a structured prompt still returns structured data; the OpenAPI validation task tries the endpoint against the live API to settle this. The end-to-end test does this outside langstar: a mise task runs a short Python script that pulls the imported prompt with `include_model=True`, invokes it, and checks that the result matches the schema. The script needs a model provider API key, which codekiln has to create and add to fnox and CI before that test can run. + +### Relationship to #728 + +[#728](https://github.com/codekiln/langstar/issues/728) splits prompt text, model and structured output into separate `create`, `get` and `update` commands for editing one field at a time. Export and import handle the whole prompt at once. Both use the same SDK calls, and once #728 lands, `prompt get --all` shows the same fields an export file holds. + +## Risks / Trade-offs + +- [LangSmith may store a pushed model differently from the source] → Import reads the prompt back and reports any difference, and the OpenAPI validation task checks the round trip before the commands are built. +- [Pushing a manifest identical to the latest commit may return 409] → Import compares first and pushes nothing when the prompt matches. +- [Most assistant integration tests are ignored today, because GitHub-built test deployments have no URL] → The test fixture task makes a deployment langstar can reach before the assistant tests are written. +- [A tag in the destination may already point at another commit] → `--update` moves the tag, and `--dry-run` lists the move. + +## Resolved Questions + +None. + +## Open Questions + +### 1 - Should compare, verify and import exit with 2 when they find a difference? + +> Context from the drafting agent, for question 1. +> The draft uses 0 for a match, 2 for a difference or failed check, and 1 for every other error, as in every other langstar command. `diff` and `git diff --exit-code` use 1 for differences, which would make a difference look like an error to scripts written against langstar. Recommendation: keep 0, 2 and 1. + + + +### 2 - Should import record destination IDs in a separate file next to the export? + +> Context from the drafting agent, for question 2. +> The draft writes `.destinations.json` and leaves the export unchanged. Keeping the record inside the export file would make one file per resource, but export would then have to keep the record when it overwrites the file. Recommendation: the separate file. + + + +### 3 - How should the end-to-end test show that an imported structured prompt still returns structured data? + +> Context from the drafting agent, for question 3. +> The LangSmith OpenAPI spec's only prompt-invoke endpoint, `POST /api/v1/prompts/invoke_prompt`, takes message templates and inputs with no model or schema, and no SDK calls it. The validation task tries it live. The draft runs a Python script through a mise task, using the LangSmith Python SDK and LangChain, which is how most users run a prompt. It needs a model provider key, such as `OPENAI_API_KEY`, which codekiln would create and add to fnox and CI. Building the model call into langstar as `prompt verify --invoke` would need langstar to call each provider's API itself. Recommendation: the Python script, with an OpenAI key and a small model. + + + +### 4 - Should the user name an assistant's prompt references with --prompt-ref? + +> Context from the drafting agent, for question 4. +> A graph reads its prompt names from whatever configuration key its author chose, so langstar cannot find them reliably. Guessing from strings that look like prompt handles would miss some and flag others. Recommendation: `--prompt-ref `, repeatable, recorded in the export file so import and verify reuse it. + + + +### 5 - Which second workspace and second deployment should the tests use? + +> Context from the drafting agent, for question 5. +> The tests need a destination workspace and a destination deployment. The draft adds `LANGSTAR_TEST_DEST_WORKSPACE_ID`, a second workspace in the test organization so the existing API key works, and a second test deployment built by the existing `TestDeploymentConfig` fixture under its own name prefix. A second deployment costs build time and hosting in every CI run that creates it. Recommendation: a second workspace in the same organization, and one long-lived destination deployment that the tests reuse. + + + +### 6 - Should prompt import refuse to run without --workspace-id even when LANGSMITH_WORKSPACE_ID is set? + +> Context from the drafting agent, for question 6. +> Issue #807 asks for an explicit destination workspace. Requiring the flag means a leftover environment variable can never send a prompt to the wrong workspace, at the cost of one more flag in scripts. Assistant import already requires `--deployment`. Recommendation: require the flag. + + diff --git a/openspec/changes/export-import-prompts-and-assistants/proposal.md b/openspec/changes/export-import-prompts-and-assistants/proposal.md new file mode 100644 index 000000000..f7a42547a --- /dev/null +++ b/openspec/changes/export-import-prompts-and-assistants/proposal.md @@ -0,0 +1,43 @@ +# Export, import, compare and verify prompts and assistants + +Issue: [#807 Export, import, compare, and verify prompts and assistants across LangSmith environments](https://github.com/codekiln/langstar/issues/807) + +## Why + +Moving an application to a new LangSmith workspace and a new deployment means recreating its prompts and its assistants there. Today a user does that by hand in the web interface, or with `langstar prompt push`, which builds a single-message prompt and drops the model, the system message and any placeholders. Nothing checks afterwards that the copy matches the original or that it can still run. + +A prompt belongs to a workspace and an assistant belongs to a deployment, but both need the same four operations: write the definition to a file that can live in Git, recreate it somewhere else, compare a copy with that file, and check that the copy has what it needs to run. + +## What Changes + +- `langstar prompt export ` writes one JSON file holding a saved prompt commit exactly as LangSmith returns it with the model included: message templates, input variables, output schema, structured output method, model binding, the repo's description, readme and tags, the commit tags, and where it came from. +- `langstar prompt import --workspace-id ` creates the prompt in the destination workspace, or adds a commit to an existing prompt when the user passes `--update`. `--dry-run` shows what it would do. After writing, it reads the prompt back and reports any difference. +- `langstar prompt compare ` reports the differences between the file and a prompt in a workspace. `langstar prompt verify ` also checks what the prompt needs in that workspace: the secrets its model reads and a model configuration for its provider. +- `langstar assistant export`, `import`, `compare` and `verify` do the same for assistants on a deployment. The file holds the name, description, graph ID, configuration, context, metadata and any prompt names the assistant refers to. `verify` checks that the destination deployment has the graph and that the destination workspace has the referenced prompts. +- Import records each destination ID (the new commit hash, or the new assistant ID) in a separate file next to the export, so importing the same file again finds the copy it made earlier instead of creating another one. +- `compare`, `verify` and `import` share one exit status scheme and print a JSON report with `-f json`. + +## Capabilities + +### New Capabilities + +- `prompt`: the `langstar prompt` command group. This change adds its export, import, compare and verify requirements; the existing prompt subcommands get their requirements when [#728 Redesign prompt create/get/update commands](https://github.com/codekiln/langstar/issues/728) lands. + +### Modified Capabilities + +- `assistant`: adds export, import, compare and verify requirements to `langstar assistant`. + +## Impact + +- SDK: `sdk/src/prompts.rs` gains a commit read that asks for the model and calls for prompt tags. `sdk/src/assistants.rs` gains the `context` field, and `assistant_id` and `if_exists` on create. +- CLI: new subcommands in `cli/src/commands/prompt.rs` and `cli/src/commands/assistant.rs`, and a shared module for the export file, the destination record, comparison and the exit status. +- Tests need a second workspace and a second deployment, and one model provider key for the structured output check. codekiln creates each one; langstar's tests then read them through `fnox.toml` and the CI workflow. +- Docs: a new usage page on moving prompts and assistants between environments. + +## Citations + +- [My/Principle/Declarative over Imperative](https://github.com/codekiln/logseq-encode-garden/blob/main/pages/My___Principle___Declarative%20over%20Imperative.md) +- [My/Principle/Explicit is better than Implicit](https://github.com/codekiln/logseq-encode-garden/blob/main/pages/My___Principle___Explicit%20is%20better%20than%20Implicit.md) +- [My/Principle/Simplify/Create Uniform Interfaces](https://github.com/codekiln/logseq-encode-garden/blob/main/pages/My___Principle___Simplify___Create%20Uniform%20Interfaces.md) +- [My/Principle/Make the Right Thing Easy and the Wrong Thing Hard](https://github.com/codekiln/logseq-encode-garden/blob/main/pages/My___Principle___Make%20the%20Right%20Thing%20Easy%20and%20the%20Wrong%20Thing%20Hard.md) +- [My/Principle/Simplify/Prefer Standards and Defaults](https://github.com/codekiln/logseq-encode-garden/blob/main/pages/My___Principle___Simplify___Prefer%20Standards%20and%20Defaults.md) diff --git a/openspec/changes/export-import-prompts-and-assistants/references/langgraph-sdk.md b/openspec/changes/export-import-prompts-and-assistants/references/langgraph-sdk.md new file mode 100644 index 000000000..a5d20229e --- /dev/null +++ b/openspec/changes/export-import-prompts-and-assistants/references/langgraph-sdk.md @@ -0,0 +1,13 @@ +# LangGraph Python SDK assistants client + +Read at [langgraph 8a77332d](https://github.com/langchain-ai/langgraph/tree/8a77332d0db75c49d296b1f3833a3a5ccd5f0dc8). + +[`create`](https://github.com/langchain-ai/langgraph/blob/8a77332d0db75c49d296b1f3833a3a5ccd5f0dc8/libs/sdk-py/langgraph_sdk/_async/assistants.py#L314) posts `graph_id`, `config`, `context`, `metadata`, `assistant_id`, `if_exists` (`raise` or `do_nothing`), `name` and `description` to `/assistants`. + +[`update`](https://github.com/langchain-ai/langgraph/blob/8a77332d0db75c49d296b1f3833a3a5ccd5f0dc8/libs/sdk-py/langgraph_sdk/_async/assistants.py#L385) patches the same fields; the server merges `metadata` into the existing metadata. [`get_versions`](https://github.com/langchain-ai/langgraph/blob/8a77332d0db75c49d296b1f3833a3a5ccd5f0dc8/libs/sdk-py/langgraph_sdk/_async/assistants.py#L656) lists the versions updates create. + +[`AssistantBase`](https://github.com/langchain-ai/langgraph/blob/8a77332d0db75c49d296b1f3833a3a5ccd5f0dc8/libs/sdk-py/langgraph_sdk/schema.py#L246) returns `assistant_id`, `graph_id`, `config`, `context`, `created_at`, `metadata`, `version`, `name` and `description`. `context` sits beside `config.configurable`; newer graphs read it and older ones read `configurable`. + +There is no endpoint that lists graphs. [`search`](https://github.com/langchain-ai/langgraph/blob/8a77332d0db75c49d296b1f3833a3a5ccd5f0dc8/libs/sdk-py/langgraph_sdk/_async/assistants.py#L528) filters by `graph_id`, and every graph has a default assistant with `metadata.created_by` set to `system`, as the [`get` docstring](https://github.com/langchain-ai/langgraph/blob/8a77332d0db75c49d296b1f3833a3a5ccd5f0dc8/libs/sdk-py/langgraph_sdk/_async/assistants.py#L64) shows. + +What langstar takes from it: the fields an assistant export carries, including `context`, and the graph check through search. diff --git a/openspec/changes/export-import-prompts-and-assistants/references/langsmith-cli.md b/openspec/changes/export-import-prompts-and-assistants/references/langsmith-cli.md new file mode 100644 index 000000000..27873927b --- /dev/null +++ b/openspec/changes/export-import-prompts-and-assistants/references/langsmith-cli.md @@ -0,0 +1,17 @@ +# LangSmith CLI (Go) + +Read at [langsmith-cli 26135038](https://github.com/langchain-ai/langsmith-cli/tree/26135038671155f05ddd725703de7bdd43c7aa9f). + +[`prompt pull`](https://github.com/langchain-ai/langsmith-cli/blob/26135038671155f05ddd725703de7bdd43c7aa9f/internal/cmd/prompt.go#L301) takes `OWNER/REPO[:COMMIT_OR_TAG]` and writes this envelope to stdout or `--output`: + +```json +{"owner": "...", "repo": "...", "commit_hash": "...", "description": "...", "manifest": {}} +``` + +It reads the commit with empty `CommitGetParams`, so the model is never requested. + +[`prompt push`](https://github.com/langchain-ai/langsmith-cli/blob/26135038671155f05ddd725703de7bdd43c7aa9f/internal/cmd/prompt.go#L359) reads the bare manifest from `--file` or stdin and sends it as `manifest`, with optional `--parent-commit` and `--description`. It does not create the repo and has no tag flag. A pull envelope fed to push is sent as the manifest itself. + +[`resolveClientOptions`](https://github.com/langchain-ai/langsmith-cli/blob/26135038671155f05ddd725703de7bdd43c7aa9f/internal/cmd/root.go#L161) picks one workspace per run from `--workspace`, `LANGSMITH_WORKSPACE_ID`, `LANGSMITH_TENANT_ID` or the profile, highest first. + +What langstar takes from it: a file between a read run and a write run, a `:tag` or `--commit` to choose the commit, and output to stdout or a file. diff --git a/openspec/changes/export-import-prompts-and-assistants/references/langsmith-go.md b/openspec/changes/export-import-prompts-and-assistants/references/langsmith-go.md new file mode 100644 index 000000000..c8b7c4707 --- /dev/null +++ b/openspec/changes/export-import-prompts-and-assistants/references/langsmith-go.md @@ -0,0 +1,13 @@ +# LangSmith Go SDK + +Read at [langsmith-go 59bb42c3](https://github.com/langchain-ai/langsmith-go/tree/59bb42c30052067fe488f9bb8370df9037c715ef). + +[`CommitGetParams`](https://github.com/langchain-ai/langsmith-go/blob/59bb42c30052067fe488f9bb8370df9037c715ef/commit.go#L337) takes `include`, a comma-separated list of `model` and `is_draft`, and marks the boolean `include_model` deprecated. + +[`CommitGetResponse`](https://github.com/langchain-ai/langsmith-go/blob/59bb42c30052067fe488f9bb8370df9037c715ef/commit.go#L262) returns `commit_hash`, `description`, `examples`, `is_draft`, `manifest`, `model_config` and `model_provider`. + +[`CommitNewParams`](https://github.com/langchain-ai/langsmith-go/blob/59bb42c30052067fe488f9bb8370df9037c715ef/commit.go#L324) sends `manifest`, `parent_commit`, `description` and `skip_webhooks`. There is no field for the model outside the manifest. + +[`RepoNewParams`](https://github.com/langchain-ai/langsmith-go/blob/59bb42c30052067fe488f9bb8370df9037c715ef/repo.go#L310) requires `repo_handle` and `is_public` and takes `description`, `readme` and `tags`. + +What langstar takes from it: `include=model` on export, and the repo fields an export carries. diff --git a/openspec/changes/export-import-prompts-and-assistants/references/langsmith-sdk.md b/openspec/changes/export-import-prompts-and-assistants/references/langsmith-sdk.md new file mode 100644 index 000000000..c4b6202be --- /dev/null +++ b/openspec/changes/export-import-prompts-and-assistants/references/langsmith-sdk.md @@ -0,0 +1,13 @@ +# LangSmith Python SDK + +Read at [langsmith-sdk 47c553ca](https://github.com/langchain-ai/langsmith-sdk/tree/47c553caf08c1989e2be3ff1886da3eb801f709f). + +[`pull_prompt_commit`](https://github.com/langchain-ai/langsmith-sdk/blob/47c553caf08c1989e2be3ff1886da3eb801f709f/python/langsmith/client.py#L10107) adds `?include_model=true` when asked. The result, [`PromptCommit`](https://github.com/langchain-ai/langsmith-sdk/blob/47c553caf08c1989e2be3ff1886da3eb801f709f/python/langsmith/schemas.py#L1035), holds the manifest plus `model_config` and `model_provider`. + +With the model included, the manifest is a `RunnableSequence`. Its `first` step is the `ChatPromptTemplate` or `StructuredPrompt` (with `schema_`), and its `last` step is a `RunnableBinding` around the chat model constructor and its settings. The [integration fixture `prompt_with_model`](https://github.com/langchain-ai/langsmith-sdk/blob/47c553caf08c1989e2be3ff1886da3eb801f709f/python/tests/integration_tests/test_prompts.py#L48) shows one. + +The chat model refers to its API key as a secret reference, `{"lc": 1, "type": "secret", "id": ["OPENAI_API_KEY"]}`. [`_manifest_has_secrets`](https://github.com/langchain-ai/langsmith-sdk/blob/47c553caf08c1989e2be3ff1886da3eb801f709f/python/langsmith/client.py#L434) finds these by matching objects whose keys are exactly `lc`, `type` and `id`. + +[`create_commit`](https://github.com/langchain-ai/langsmith-sdk/blob/47c553caf08c1989e2be3ff1886da3eb801f709f/python/langsmith/client.py#L9915) resolves a missing parent to the latest commit, posts `{parent_commit, manifest, description}`, then posts each tag to `/repos/{owner}/{repo}/tags`. [`push_prompt`](https://github.com/langchain-ai/langsmith-sdk/blob/47c553caf08c1989e2be3ff1886da3eb801f709f/python/langsmith/client.py#L10309) creates the repo first when it does not exist. + +What langstar takes from it: the shape of a manifest with its model, how to find secret names, and the order of create repo, commit with parent, then tags. diff --git a/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md b/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md new file mode 100644 index 000000000..fface00d1 --- /dev/null +++ b/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md @@ -0,0 +1,122 @@ +## ADDED Requirements + +### Requirement: Export an assistant to a file + +`langstar assistant export --deployment ` SHALL write one JSON file holding the assistant's name, description, graph ID, configuration, context and metadata, and a `source` object naming the deployment, assistant ID and version. `--output ` SHALL choose the file; without it the file goes to stdout. Exporting the same assistant version twice SHALL produce identical files. + +#### Scenario: Export an assistant + +- **WHEN** a user runs `langstar assistant export --deployment my-agent --output support-bot.json` +- **THEN** `support-bot.json` holds the assistant's name, graph ID, configuration, context and metadata +- **AND** `source.assistant_id` holds the source assistant's ID + +#### Scenario: Assistant not found + +- **WHEN** the deployment has no assistant with that ID +- **THEN** langstar exits with status 1 and an error naming the assistant ID + +### Requirement: Record the prompts an assistant refers to + +`langstar assistant export` SHALL accept `--prompt-ref `, repeatable, naming a string in the assistant's configuration or context that holds a prompt name. The file SHALL list each pointer and the prompt name it holds under `requires.prompts`. A pointer that resolves to no string SHALL be an error. + +#### Scenario: Assistant names its system prompt + +- **WHEN** a user runs `langstar assistant export --deployment my-agent --prompt-ref /config/configurable/system_prompt` +- **AND** that value is `support-system:prod` +- **THEN** `requires.prompts` holds the pointer and `support-system:prod` + +#### Scenario: Pointer names nothing + +- **WHEN** the pointer resolves to no value in the assistant +- **THEN** langstar exits with status 1 and an error naming the pointer + +### Requirement: Import an assistant into a deployment + +`langstar assistant import --deployment ` SHALL create an assistant on the deployment with the file's name, description, graph ID, configuration, context and metadata, and print the new assistant ID. It SHALL record the deployment and the new assistant ID in the destination record file, `.destinations.json` unless `--record ` names another. + +#### Scenario: First import + +- **WHEN** a user runs `langstar assistant import support-bot.json --deployment my-agent-prod` +- **AND** the destination record has no entry for `my-agent-prod` +- **THEN** langstar creates the assistant and prints its new ID +- **AND** `support-bot.destinations.json` holds `my-agent-prod` with that ID + +### Requirement: Find the earlier copy instead of creating another + +When the destination record holds an assistant ID for the deployment, `langstar assistant import` SHALL compare the file with that assistant and create nothing. It SHALL exit with status 0 when they match, and list the differences and exit with status 2 when they differ, unless the user passes `--update`. When the recorded assistant no longer exists, it SHALL exit with status 1 and say so. + +#### Scenario: Import the same file again + +- **WHEN** a user imports `support-bot.json` into `my-agent-prod` a second time with no changes +- **THEN** langstar creates no assistant and reports that the assistant is unchanged + +#### Scenario: Recorded assistant was deleted + +- **WHEN** the recorded assistant ID no longer exists on the deployment +- **THEN** langstar exits with status 1 and names the missing assistant ID + +### Requirement: Update an existing assistant only on request + +`langstar assistant import --update` SHALL update the recorded assistant, or the assistant named by `--assistant-id `, with the file's name, description, graph ID, configuration, context and metadata. `--assistant-id` SHALL take precedence over the destination record. + +#### Scenario: Update the recorded assistant + +- **WHEN** the file's configuration differs from the recorded assistant +- **AND** the user runs `langstar assistant import support-bot.json --deployment my-agent-prod --update` +- **THEN** langstar updates that assistant and creates no other + +#### Scenario: Update an assistant chosen by ID + +- **WHEN** the user runs `langstar assistant import support-bot.json --deployment my-agent-prod --update --assistant-id ` +- **THEN** langstar updates that assistant and records its ID for `my-agent-prod` + +### Requirement: Dry run an assistant import + +`langstar assistant import --dry-run` SHALL report whether it would create the assistant, update it, or leave it unchanged, list the differences, and change nothing on the deployment or in the destination record. + +#### Scenario: Dry run of a first import + +- **WHEN** a user runs `langstar assistant import support-bot.json --deployment my-agent-prod --dry-run` +- **THEN** langstar reports that it would create the assistant and creates nothing + +### Requirement: Read the assistant back after import + +After writing, `langstar assistant import` SHALL read the assistant back from the deployment and compare it with the file. When they differ it SHALL list the differences and exit with status 2. + +#### Scenario: Deployment drops a configuration key + +- **WHEN** the assistant read back lacks a configuration key the file holds +- **THEN** langstar lists that key as a difference and exits with status 2 + +### Requirement: Compare an assistant with its export file + +`langstar assistant compare --deployment ` SHALL compare the file with the recorded assistant, or the one named by `--assistant-id`, and list each difference by its JSON path with both values. Fields the server sets, such as the assistant ID, version and timestamps, SHALL be left out of the comparison. + +#### Scenario: Metadata changed on the deployment + +- **WHEN** someone changed a metadata value on the deployment's assistant +- **THEN** langstar lists that metadata path with both values and exits with status 2 + +### Requirement: Verify that an assistant can run on a deployment + +`langstar assistant verify --deployment ` SHALL do what `compare` does and also check that the deployment serves the file's graph ID and that the workspace, from `--workspace-id` or the configured workspace, has each prompt under `requires.prompts`. It SHALL name each missing graph and prompt and exit with status 2 when any check fails. + +#### Scenario: Graph missing on the destination + +- **WHEN** the destination deployment serves no graph with the file's graph ID +- **THEN** langstar reports the missing graph ID and exits with status 2 + +#### Scenario: Referenced prompt missing + +- **WHEN** the file lists `support-system:prod` under `requires.prompts` +- **AND** the workspace has no `support-system` prompt with a `prod` tag +- **THEN** langstar reports `support-system:prod` as missing and exits with status 2 + +### Requirement: Assistant transfer results share the prompt transfer shape + +`assistant compare`, `assistant verify` and `assistant import` SHALL use the exit statuses and the JSON report shape of `prompt compare`, `prompt verify` and `prompt import`. + +#### Scenario: Machine-readable verify + +- **WHEN** a user runs `langstar -f json assistant verify support-bot.json --deployment my-agent-prod` +- **THEN** stdout holds one JSON object with `status`, `differences` and `checks` diff --git a/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md new file mode 100644 index 000000000..ae12313fb --- /dev/null +++ b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md @@ -0,0 +1,166 @@ +## Purpose + +`langstar prompt` manages the prompts saved in a LangSmith workspace, including writing a prompt to a file and recreating it in another workspace. + +## ADDED Requirements + +### Requirement: Export a prompt commit to a file + +`langstar prompt export ` SHALL write one JSON file holding the prompt's repo details (handle, description, readme, tags, visibility), the commit's manifest as LangSmith returns it with the model included, the commit's model configuration and provider, the commit tags that point at it, and a `source` object naming the workspace, handle and commit hash. `--output ` SHALL choose the file; without it the file goes to stdout. + +#### Scenario: Export the latest commit + +- **WHEN** a user runs `langstar prompt export my-prompt --output my-prompt.json` +- **THEN** langstar writes `my-prompt.json` with the manifest of the latest commit, including the model binding +- **AND** `source.commit_hash` holds that commit's hash + +#### Scenario: Export a structured prompt + +- **WHEN** the prompt is a structured prompt with a nested JSON Schema and the `function_calling` method +- **THEN** the exported manifest holds the schema and the method unchanged + +#### Scenario: Prompt not found + +- **WHEN** no prompt in the workspace has the handle +- **THEN** langstar exits with status 1 and an error naming the handle + +### Requirement: Export a chosen commit or tag + +`langstar prompt export` SHALL accept `--commit ` and export that commit. Without the flag it SHALL export the latest commit. + +#### Scenario: Export a tagged commit + +- **WHEN** a user runs `langstar prompt export my-prompt --commit prod --output my-prompt.json` +- **THEN** the file holds the commit the `prod` tag points at + +### Requirement: Write the same file for the same commit + +Exporting the same commit twice SHALL produce identical files. The file SHALL hold no export time, and langstar SHALL write object keys in a fixed order. + +#### Scenario: Export twice + +- **WHEN** a user exports the same commit twice to two files +- **THEN** the two files are identical + +### Requirement: List the credentials a prompt needs without their values + +The export file SHALL list, under `requires.secrets`, the name of every secret the manifest refers to, and under `requires.model_provider`, the model's provider. It SHALL hold no secret values. + +#### Scenario: Model reads an API key secret + +- **WHEN** the prompt's model refers to the secret `OPENAI_API_KEY` +- **THEN** `requires.secrets` holds `OPENAI_API_KEY` +- **AND** no value of that secret appears in the file + +### Requirement: Import names its destination workspace + +`langstar prompt import ` SHALL require `--workspace-id`. It SHALL refuse to run when the flag is missing, even when `LANGSMITH_WORKSPACE_ID` is set. + +#### Scenario: Missing destination workspace + +- **WHEN** a user runs `langstar prompt import my-prompt.json` without `--workspace-id` +- **THEN** langstar exits with status 1 and an error asking for `--workspace-id` + +### Requirement: Import a prompt that does not exist yet + +When the destination workspace has no prompt with the file's handle, `langstar prompt import` SHALL create the prompt with the file's repo details, push the file's manifest as its first commit, and create the file's commit tags on that commit. + +#### Scenario: First import + +- **WHEN** a user runs `langstar prompt import my-prompt.json --workspace-id ` +- **AND** the destination has no prompt named `my-prompt` +- **THEN** langstar creates `my-prompt` in the destination with the file's manifest, model and tags +- **AND** prints the new commit hash + +### Requirement: Change an existing prompt only with --update + +When the destination prompt exists and its latest commit differs from the file, `langstar prompt import` SHALL change nothing and exit with status 2, listing the differences, unless the user passes `--update`. With `--update` it SHALL push the file's manifest as a new commit whose parent is the destination's latest commit, and move the file's tags to it. + +#### Scenario: Destination differs and --update is missing + +- **WHEN** the destination prompt's latest commit differs from the file +- **AND** the user runs `langstar prompt import` without `--update` +- **THEN** langstar pushes nothing, lists the differences and exits with status 2 + +#### Scenario: Update an existing prompt + +- **WHEN** the user runs the same import with `--update` +- **THEN** langstar adds a commit to the destination prompt holding the file's manifest + +#### Scenario: Destination already matches + +- **WHEN** the destination prompt's latest commit matches the file +- **THEN** langstar pushes nothing, reports that the prompt is unchanged and exits with status 0 + +### Requirement: Record the destination commit + +After a successful import, langstar SHALL record the destination workspace, handle and commit hash in a destination record file, `.destinations.json` next to the export unless `--record ` names another. Each import SHALL add or replace only the entry for its own destination workspace. + +#### Scenario: Import into two workspaces + +- **WHEN** a user imports `my-prompt.json` into a staging workspace and then into a production workspace +- **THEN** `my-prompt.destinations.json` holds one entry for each workspace with its own commit hash +- **AND** `my-prompt.json` is unchanged + +### Requirement: Warn when the destination changed since the last import + +When the destination record holds a commit for the destination workspace and the destination's latest commit is a different one, `langstar prompt import` SHALL say that someone changed the prompt after the last import, in addition to listing the differences. + +#### Scenario: Someone edited the destination prompt + +- **WHEN** a user edited the destination prompt in the web interface after the last import +- **AND** runs `langstar prompt import my-prompt.json --workspace-id ` +- **THEN** langstar reports that the destination changed since the last recorded import and exits with status 2 + +### Requirement: Dry run an import + +`langstar prompt import --dry-run` SHALL report whether it would create the prompt, add a commit, or leave it unchanged, list the differences, and change nothing in the workspace or the destination record. + +#### Scenario: Dry run against a new workspace + +- **WHEN** a user runs `langstar prompt import my-prompt.json --workspace-id --dry-run` +- **AND** the destination has no prompt named `my-prompt` +- **THEN** langstar reports that it would create the prompt and creates nothing + +### Requirement: Read the prompt back after import + +After writing, `langstar prompt import` SHALL read the destination commit back with the model included and compare it with the file. When they differ it SHALL list the differences and exit with status 2. + +#### Scenario: LangSmith drops a field + +- **WHEN** the destination commit read back after import lacks a field the file holds +- **THEN** langstar lists that field as a difference and exits with status 2 + +### Requirement: Compare a prompt with its export file + +`langstar prompt compare ` SHALL read the prompt with the file's handle from the workspace, with the model included, and list each difference by its JSON path with the file's value and the workspace's value. `--commit ` SHALL choose the commit; without it langstar SHALL read the latest commit. Fields the server sets, such as IDs and timestamps, SHALL be left out of the comparison. + +#### Scenario: Prompt matches + +- **WHEN** the workspace's prompt matches the file +- **THEN** langstar reports no differences and exits with status 0 + +#### Scenario: Model temperature changed + +- **WHEN** someone changed the model temperature in the workspace's prompt +- **THEN** langstar lists the temperature's JSON path with both values and exits with status 2 + +### Requirement: Verify that a prompt can run in a workspace + +`langstar prompt verify ` SHALL do what `compare` does and also check that the workspace has every secret in `requires.secrets` and a model configuration for `requires.model_provider`. It SHALL name each missing secret and provider and exit with status 2 when any check fails. + +#### Scenario: Secret missing in the destination + +- **WHEN** the destination workspace has no secret named `OPENAI_API_KEY` +- **AND** the file lists `OPENAI_API_KEY` under `requires.secrets` +- **THEN** langstar reports `OPENAI_API_KEY` as missing and exits with status 2 + +### Requirement: Report results in one shape with one set of exit statuses + +`prompt compare`, `prompt verify` and `prompt import` SHALL exit with status 0 when the prompt matches and every check passes, 2 when they find a difference or a failed check, and 1 on any other error. With `-f json` each SHALL print one JSON object with `status`, `differences` and `checks`. + +#### Scenario: Machine-readable compare + +- **WHEN** a user runs `langstar -f json prompt compare my-prompt.json` +- **THEN** stdout holds one JSON object with `status`, `differences` and `checks` +- **AND** progress messages go to stderr diff --git a/openspec/changes/export-import-prompts-and-assistants/tasks.md b/openspec/changes/export-import-prompts-and-assistants/tasks.md new file mode 100644 index 000000000..9f3cf8893 --- /dev/null +++ b/openspec/changes/export-import-prompts-and-assistants/tasks.md @@ -0,0 +1,66 @@ +# Tasks + +Each numbered group is one pull request, in the order listed. + +## 1. Check the API round trip before building on it + +- [ ] 1.1 Extract the commit get, commit create, repo tags, workspace secrets and playground settings endpoints from `reference/openapi/langchain/langsmith/openapi.json` into `reference/api-specs/langsmith/` and add their jq queries to `FRAGMENTS.md`; verify by rerunning each query and getting the same file +- [ ] 1.2 Write `reference/research/807-openapi-validation.md`, recording from the live API: the manifest returned with `include=model` for a structured prompt with a model, whether pushing that manifest to a second workspace returns the same model, what pushing an identical manifest returns, what the Agent Server returns for `context`, and what `POST /api/v1/prompts/invoke_prompt` returns for a prompt with a model; verify that every claim in the report names the request that produced it + +## 2. Test environments + +- [ ] 2.1 Add the destination workspace and the model provider key to `fnox.toml`, the CI workflow and `docs/dev/environment-variables.md`, with placeholders only, after codekiln has created the workspace and the key and stored them in 1Password and the repo's CI secrets; verify with `mise run check` and a CI run that reads both +- [ ] 2.2 Add a destination deployment fixture to `sdk/src/test_utils.rs` and `cli/tests/common/fixtures.rs` that returns a deployment with a reachable URL; verify with an integration test that lists assistants on it + +## 3. SDK reads and writes the fields a transfer needs + +- [ ] 3.1 Add a commit read that sends `include=model` and returns `model_config` and `model_provider` to `sdk/src/prompts.rs`; verify with an httpmock test that asserts the query string and the parsed fields, and an integration test that reads a prompt with a model +- [ ] 3.2 Add listing and creating prompt tags to `sdk/src/prompts.rs`; verify with httpmock tests and an integration test that creates a tag and reads it back through the SDK +- [ ] 3.3 Add `context` to `Assistant`, `CreateAssistantRequest` and `UpdateAssistantRequest`, and `assistant_id` and `if_exists` to `CreateAssistantRequest`, in `sdk/src/assistants.rs`; verify with httpmock tests of the request bodies and an integration test that creates an assistant with context and reads it back + +## 4. Shared transfer module + +- [ ] 4.1 Add the export file types, with sorted keys and no export time, and the destination record file, to a new module in `cli/src/`; verify with unit tests that serialize the same value twice to identical text and that updating one destination leaves the others unchanged +- [ ] 4.2 Add the JSON comparison that drops server-set fields and reports differences by JSON Pointer; verify with unit tests on nested objects, arrays, missing keys and ignored fields +- [ ] 4.3 Add the shared result report and the exit statuses 0, 2 and 1, printing JSON to stdout and progress to stderr; verify with unit tests of the report and a CLI test that checks the exit status +- [ ] 4.4 Add the secret reference walk that lists secret names from a manifest; verify with unit tests on the manifest fixtures from `references/langsmith-sdk.md` + +## 5. prompt export + +- [ ] 5.1 Add `langstar prompt export` with `--commit`, `--output` and the scoping flags; verify with integration tests that create prompts through the SDK (a `json_schema` prompt, a `function_calling` prompt with a nested schema, a prompt with several messages and a placeholder, and a prompt with a model and metadata), export each through the CLI, and assert every field of the file against the SDK's read of the commit +- [ ] 5.2 Document `prompt export` in `docs/usage/` and the README; verify that each documented command runs as written + +## 6. prompt compare and verify + +- [ ] 6.1 Add `langstar prompt compare`; verify with integration tests that export a prompt, change its model temperature through the SDK, and assert the reported path, both values and exit status 2 +- [ ] 6.2 Add `langstar prompt verify` with the secret and model configuration checks; verify with integration tests against a workspace that lacks a listed secret and one that has it +- [ ] 6.3 Document compare and verify; verify that each documented command runs as written + +## 7. prompt import + +- [ ] 7.1 Add `langstar prompt import` with required `--workspace-id`, creating the repo, the commit and its tags, then reading the commit back; verify with integration tests that export from the test workspace, import into the destination workspace, and compare the full source and destination commits through the SDK with the model included, for every prompt shape from 5.1 +- [ ] 7.2 Add `--update`, `--dry-run`, `--record` and the destination record; verify with integration tests that a repeated import creates no commit, that a changed destination is refused without `--update`, that `--update` adds one commit with the right parent, and that `--dry-run` writes nothing +- [ ] 7.3 Add the structured output check as a mise task that pulls the imported prompt with `include_model=True`, invokes it and validates the result against the schema; verify by running it from the import integration test for both methods +- [ ] 7.4 Document prompt import, the workspace scope of prompts and setting up destination credentials; verify that each documented command runs as written + +## 8. assistant export + +- [ ] 8.1 Add `langstar assistant export` with `--output` and `--prompt-ref`; verify with integration tests that create an assistant with configuration, context and metadata through the SDK and assert every field of the exported file, and an error test for a pointer that names nothing +- [ ] 8.2 Document assistant export; verify that each documented command runs as written + +## 9. assistant compare and verify + +- [ ] 9.1 Add `langstar assistant compare`; verify with an integration test that changes metadata through the SDK and asserts the reported path and exit status 2 +- [ ] 9.2 Add `langstar assistant verify` with the graph and prompt checks; verify with integration tests for a missing graph ID and a missing prompt tag +- [ ] 9.3 Document assistant compare and verify; verify that each documented command runs as written + +## 10. assistant import + +- [ ] 10.1 Add `langstar assistant import` with creation, read-back and the destination record; verify with an integration test that imports into the destination deployment and checks name, graph ID, configuration, context, metadata and prompt references through the Agent Server API +- [ ] 10.2 Add `--update`, `--assistant-id` and `--dry-run`; verify with integration tests that a repeated import creates no assistant, that `--update` changes the recorded one, and that a deleted recorded assistant exits with status 1 +- [ ] 10.3 Add an end-to-end test that imports prompts into the destination workspace and then an assistant that refers to them onto the destination deployment, and runs `assistant verify`; verify that it passes and that it fails after deleting one prompt tag +- [ ] 10.4 Document assistant import, the deployment scope of assistants, and how export and import relate to #728's `create`, `get` and `update`; verify that each documented command runs as written + +## Workflow follow-up + +- Run `openspec validate export-import-prompts-and-assistants --strict`, then archive the change in the pull request for group 10. From 828864dac207bec43aac5f60c16d4cd2bc2f6d55 Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Tue, 13 Oct 2026 18:11:09 -0400 Subject: [PATCH 02/90] =?UTF-8?q?=F0=9F=93=9A=20docs(openspec):=20fix=20th?= =?UTF-8?q?e=20claims=20and=20headings=20codekiln-review=20flagged=20on=20?= =?UTF-8?q?the=20transfer=20design?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add the SDK fields and the deployment URL fix the assistant commands need (tasks 2.3 and 3.3), check a prompt tag through the commits endpoint, match the model provider by class name and report a missing saved model configuration as a note (Open Question 7), and turn topic headings into decisions. #807 Export, import, compare, and verify prompts and assistants across LangSmith environments Co-Authored-By: Claude Opus 5.5 --- .../design.md | 43 +++++++++++-------- .../proposal.md | 12 +++--- .../specs/assistant/spec.md | 2 +- .../specs/prompt/spec.md | 13 ++++-- .../tasks.md | 15 ++++--- 5 files changed, 50 insertions(+), 35 deletions(-) diff --git a/openspec/changes/export-import-prompts-and-assistants/design.md b/openspec/changes/export-import-prompts-and-assistants/design.md index bc65b0547..0379fbe02 100644 --- a/openspec/changes/export-import-prompts-and-assistants/design.md +++ b/openspec/changes/export-import-prompts-and-assistants/design.md @@ -2,9 +2,9 @@ ## Context -`langstar prompt push` and `pull` in `cli/src/commands/prompt.rs` handle one shape of prompt: a single human message, with an optional schema from [#402 structured output prompts](https://github.com/codekiln/langstar/issues/402). `PromptClient` in `sdk/src/prompts.rs` keeps the manifest as raw JSON and has a typed `StructuredPrompt` view that holds messages, the schema and the method. No call asks LangSmith for the model, so neither command sees one. +`langstar prompt push` in `cli/src/commands/prompt.rs` builds one shape of prompt: a single human message, with an optional schema from [#402 structured output prompts](https://github.com/codekiln/langstar/issues/402). `langstar prompt pull` prints any manifest LangSmith returns, but `PromptClient` in `sdk/src/prompts.rs` never asks LangSmith for the model, so that manifest has none. `PromptClient` keeps the manifest as raw JSON and has a typed `StructuredPrompt` view that holds messages, the schema and the method. -`langstar assistant` in `cli/src/commands/assistant.rs` takes `--deployment` on every subcommand and sends requests to that deployment's Agent Server. `AssistantClient` in `sdk/src/assistants.rs` has no `context` field, and its create request cannot set `assistant_id` or `if_exists`. +`langstar assistant` in `cli/src/commands/assistant.rs` takes `--deployment` on every subcommand and sends requests to that deployment's Agent Server. `AssistantClient` in `sdk/src/assistants.rs` has no `context` or `version` field, its create request cannot set `assistant_id` or `if_exists`, and its search cannot filter by graph ID. Langstar reads one API key and one workspace at a time, from flags, environment variables or the config file. Every error exits with status 1. @@ -20,15 +20,15 @@ Langstar reads one API key and one workspace at a time, from flags, environment - Copy a prompt or assistant between environments in one command. - Export or import every prompt in a workspace, or every assistant on a deployment, in one run. -- Redesign `prompt create`, `get` and `update`, which [#728 Redesign prompt commands around create/get/update](https://github.com/codekiln/langstar/issues/728) covers. +- Redesign `prompt create`, `get` and `update`, which [#728 Implement CRUD command structure (create, get, update) for prompt commands](https://github.com/codekiln/langstar/issues/728) covers. ## Decisions -### A file carries each resource between environments +### Export writes a file with the source's credentials, and import reads it with the destination's Export runs with the source environment's credentials and writes a file. Import runs later with the destination's credentials and reads it. Each run uses langstar's existing settings, so a user moving a prompt to another organization sets a different `LANGSMITH_API_KEY` for the import. Langstar needs no second set of credentials and no profiles. The file sits in Git between the two runs, so a reviewer sees the prompt or assistant change in a pull request. -The LangSmith Go CLI follows the same file workflow, as `references/langsmith-cli.md` shows. Langstar differs from it on purpose. Langstar's export asks for the model and the Go CLI's pull does not. Langstar's import reads the file export writes, while the Go CLI's push wants the bare manifest and rejects its own pull output. Langstar's import also requires an explicit `--workspace-id`, so the destination is named on the command line and never comes from a leftover environment variable. +The LangSmith Go CLI moves a prompt through a file the same way, as `references/langsmith-cli.md` shows. Langstar's export asks for the model, and the Go CLI's pull does not. Langstar's import reads the file export writes, while the Go CLI's push sends whatever JSON it reads as the manifest, so its own pull output reaches LangSmith wrapped in an envelope. Langstar's import also requires `--workspace-id`, so the user names the destination on the command line and a leftover environment variable never chooses it. ### The export file keeps the manifest LangSmith returned @@ -53,7 +53,7 @@ The file holds no export time and langstar sorts its keys, so exporting an uncha ### Export asks LangSmith for the model -Export reads the commit with `include=model`, the form the Go SDK uses (`references/langsmith-go.md`). The Python SDK sends the older `include_model=true`, and the LangSmith OpenAPI spec accepts both. With the model included, the manifest becomes a `RunnableSequence` whose last step binds the chat model and its settings, and the response also carries `model_config` and `model_provider` beside the manifest (`references/langsmith-sdk.md`). Creating a commit accepts only `manifest`, `parent_commit` and `description`, so the model reaches the destination inside the manifest. The OpenAPI validation task checks that the destination returns the same model after the push. +Export reads the commit with `include=model`, the form the Go SDK uses (`references/langsmith-go.md`). The Python SDK sends the older `include_model=true`, and the LangSmith OpenAPI spec accepts both. With the model included, the manifest becomes a `RunnableSequence` whose last step binds the chat model and its settings, and the response also carries `model_config` and `model_provider` beside the manifest (`references/langsmith-sdk.md`). No field of the create-commit request holds the model, so the model reaches the destination inside the manifest. [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](https://github.com/codekiln/langstar/blob/aeb831e1e048a24a6c2c9ea7761d5131fc3ad2c2/openspec/changes/export-import-prompts-and-assistants/tasks.md#L8) pushes that manifest to a second workspace and records whether LangSmith returns the same model. ### Import records what it created in a separate file @@ -69,31 +69,31 @@ Compare reads the destination, removes the fields the server sets (IDs, version ### Verify checks what the copy needs to run -Prompt verify looks up each name in `requires.secrets` among the workspace's secrets (`GET /api/v1/workspaces/current/secrets`, which `langstar secrets list` already calls) and looks for a model configuration whose provider matches `requires.model_provider` (`GET /api/v1/playground-settings`, used by `langstar model-config list`). It reports names only. Langstar finds secret names by walking the manifest for LangChain secret references, `{"lc": 1, "type": "secret", "id": ["OPENAI_API_KEY"]}`, the shape the Python SDK's `_manifest_has_secrets` matches. +Prompt verify looks up each name in `requires.secrets` among the workspace's secrets (`GET /api/v1/workspaces/current/secrets`, which `langstar secrets list` already calls). It also looks for a saved model configuration (`GET /api/v1/playground-settings`, which `langstar model-config list` calls) whose settings `id` ends in the class name that `requires.model_provider` holds, such as `ChatOpenAI`. The prompt runs without a saved model configuration, because its manifest carries the model, so verify reports a missing one as a note and leaves the exit status alone; [#807 Export, import, compare, and verify prompts and assistants across LangSmith environments](https://github.com/codekiln/langstar/issues/807) asks for the report, and Open Question 7 asks whether to keep it. Verify reports names only. Langstar finds secret names by walking the manifest for LangChain secret references, `{"lc": 1, "type": "secret", "id": ["OPENAI_API_KEY"]}`, the shape the Python SDK's `_manifest_has_secrets` matches. -Assistant verify checks the graph with `POST /assistants/search` filtered to the graph ID; every graph on a deployment has a default assistant, so an empty result means the deployment does not serve the graph. It checks each prompt in `requires.prompts` with the same lookup `langstar prompt get` uses. +Assistant verify checks the graph with `POST /assistants/search` filtered to the graph ID; every graph on a deployment has a default assistant, so an empty result means the deployment does not serve the graph. It checks each prompt in `requires.prompts`, such as `support-system:prod`, by asking LangSmith for the commit that tag points at (`GET /api/v1/commits/{owner}/{repo}/{tag}`, the request `langstar prompt pull --commit prod` already sends). A 404 means the workspace lacks the prompt or its tag. ### The user names an assistant's prompt references -An assistant refers to a prompt through whatever configuration key its graph reads, so langstar cannot find the references by itself. `assistant export --prompt-ref ` names each one. Prompts should be referred to by handle and tag, such as `support-system:prod`, because a commit hash differs between workspaces. Import carries commit tags across for this reason. +An assistant refers to a prompt through whatever configuration key its graph reads, so langstar cannot find the references by itself. `assistant export --prompt-ref ` names each one. The assistant's configuration should name each prompt by handle and tag, such as `support-system:prod`, because the same prompt gets a different commit hash in each workspace. Import copies the commit tags to the destination workspace, so `support-system:prod` finds the prompt there too. -### Exit statuses +### compare, verify and import exit with 2 when they find a difference Status 0 means the resource matches and every check passed. Status 2 means langstar found a difference or a failed check. Status 1 stays the status for every other error, as in every other langstar command. `diff` uses 1 for differences and 2 for errors; langstar keeps 1 for errors so that a script testing for failure sees the same status from every command. -### Structured output check +### A Python script checks that an imported structured prompt still returns structured data -The LangSmith OpenAPI spec has one endpoint that invokes a prompt, `POST /api/v1/prompts/invoke_prompt`. Its body, `InvokePromptPayload`, holds only message templates, `template_format` and `inputs`, with no field for a model or an output schema, and the spec gives no shape for its response. No LangSmith SDK calls it. So the API offers no documented way to show that a structured prompt still returns structured data; the OpenAPI validation task tries the endpoint against the live API to settle this. The end-to-end test does this outside langstar: a mise task runs a short Python script that pulls the imported prompt with `include_model=True`, invokes it, and checks that the result matches the schema. The script needs a model provider API key, which codekiln has to create and add to fnox and CI before that test can run. +A mise task runs a short Python script that pulls the imported prompt with `include_model=True`, invokes it, and checks that the result matches the prompt's schema. The script exists because the LangSmith API has no documented way to run a prompt with its model: the one prompt endpoint in the OpenAPI spec, `POST /api/v1/prompts/invoke_prompt`, takes message templates and inputs with no model or schema, and no LangSmith SDK calls it. [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](https://github.com/codekiln/langstar/blob/aeb831e1e048a24a6c2c9ea7761d5131fc3ad2c2/openspec/changes/export-import-prompts-and-assistants/tasks.md#L8) tries that endpoint against the live API in case it does more than the spec says. The script needs a model provider API key, which codekiln has to create and add to fnox and CI before the test can run. -### Relationship to #728 +### Export and import use the same SDK calls as the planned `prompt create`, `get` and `update` -[#728](https://github.com/codekiln/langstar/issues/728) splits prompt text, model and structured output into separate `create`, `get` and `update` commands for editing one field at a time. Export and import handle the whole prompt at once. Both use the same SDK calls, and once #728 lands, `prompt get --all` shows the same fields an export file holds. +[#728 Implement CRUD command structure (create, get, update) for prompt commands](https://github.com/codekiln/langstar/issues/728) splits prompt text, model and structured output into separate `create`, `get` and `update` commands for editing one field at a time. Export and import handle the whole prompt at once. Both use the same SDK calls, and once #728 lands, `prompt get --all` shows the same fields an export file holds. ## Risks / Trade-offs -- [LangSmith may store a pushed model differently from the source] → Import reads the prompt back and reports any difference, and the OpenAPI validation task checks the round trip before the commands are built. +- [LangSmith may store a pushed model differently from the source] → Import reads the prompt back and reports any difference, and [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](https://github.com/codekiln/langstar/blob/aeb831e1e048a24a6c2c9ea7761d5131fc3ad2c2/openspec/changes/export-import-prompts-and-assistants/tasks.md#L8) checks the round trip before the commands are built. - [Pushing a manifest identical to the latest commit may return 409] → Import compares first and pushes nothing when the prompt matches. -- [Most assistant integration tests are ignored today, because GitHub-built test deployments have no URL] → The test fixture task makes a deployment langstar can reach before the assistant tests are written. +- [langstar finds a deployment's Agent Server URL only in `source_config.custom_url` (`resolve_deployment_url` in `cli/src/deployment_utils.rs`), and a deployment built from GitHub has none, so every `langstar assistant` command, the new ones included, fails on such a deployment] → Task 2.3 changes `resolve_deployment_url` to find the URL of a GitHub-built deployment, before any assistant work in groups 8 to 10 starts. - [A tag in the destination may already point at another commit] → `--update` moves the tag, and `--dry-run` lists the move. ## Resolved Questions @@ -119,7 +119,7 @@ None. ### 3 - How should the end-to-end test show that an imported structured prompt still returns structured data? > Context from the drafting agent, for question 3. -> The LangSmith OpenAPI spec's only prompt-invoke endpoint, `POST /api/v1/prompts/invoke_prompt`, takes message templates and inputs with no model or schema, and no SDK calls it. The validation task tries it live. The draft runs a Python script through a mise task, using the LangSmith Python SDK and LangChain, which is how most users run a prompt. It needs a model provider key, such as `OPENAI_API_KEY`, which codekiln would create and add to fnox and CI. Building the model call into langstar as `prompt verify --invoke` would need langstar to call each provider's API itself. Recommendation: the Python script, with an OpenAI key and a small model. +> The LangSmith OpenAPI spec's only prompt-invoke endpoint, `POST /api/v1/prompts/invoke_prompt`, takes message templates and inputs with no model or schema, and no SDK calls it. [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](https://github.com/codekiln/langstar/blob/aeb831e1e048a24a6c2c9ea7761d5131fc3ad2c2/openspec/changes/export-import-prompts-and-assistants/tasks.md#L8) tries it live. The draft runs a Python script through a mise task, using the LangSmith Python SDK and LangChain, which is how most users run a prompt. It needs a model provider key, such as `OPENAI_API_KEY`, which codekiln would create and add to fnox and CI. Building the model call into langstar as `prompt verify --invoke` would need langstar to call each provider's API itself. Recommendation: the Python script, with an OpenAI key and a small model. @@ -133,7 +133,7 @@ None. ### 5 - Which second workspace and second deployment should the tests use? > Context from the drafting agent, for question 5. -> The tests need a destination workspace and a destination deployment. The draft adds `LANGSTAR_TEST_DEST_WORKSPACE_ID`, a second workspace in the test organization so the existing API key works, and a second test deployment built by the existing `TestDeploymentConfig` fixture under its own name prefix. A second deployment costs build time and hosting in every CI run that creates it. Recommendation: a second workspace in the same organization, and one long-lived destination deployment that the tests reuse. +> The tests need a destination workspace and a destination deployment. The draft adds `LANGSTAR_TEST_DEST_WORKSPACE_ID`, a second workspace in the test organization so the existing API key works, and a second test deployment built by the existing `TestDeploymentConfig` fixture under its own name prefix. A second deployment costs build time and hosting in every CI run that creates it. Recommendation: a second workspace in the same organization, which codekiln creates, and one destination deployment that the `TestDeploymentConfig` fixture creates on the first run and the tests reuse after that. @@ -143,3 +143,10 @@ None. > Issue #807 asks for an explicit destination workspace. Requiring the flag means a leftover environment variable can never send a prompt to the wrong workspace, at the cost of one more flag in scripts. Assistant import already requires `--deployment`. Recommendation: require the flag. + +### 7 - Should prompt verify report a missing saved model configuration? + +> Context from the drafting agent, for question 7. +> Issue #807 asks import to report missing model configurations, so the draft has prompt verify look for a saved model configuration (`GET /api/v1/playground-settings`) whose settings `id` ends in the prompt's `model_provider` class name, such as `ChatOpenAI`. Nothing fails at run time without one, because the prompt's manifest carries the model and its settings; the secret check is the one that finds a prompt that cannot authenticate. Reporting it as a failure would make verify exit with 2 for a prompt that runs. Recommendation: report a missing saved model configuration as a note that leaves the exit status alone. + + diff --git a/openspec/changes/export-import-prompts-and-assistants/proposal.md b/openspec/changes/export-import-prompts-and-assistants/proposal.md index f7a42547a..43f3b4970 100644 --- a/openspec/changes/export-import-prompts-and-assistants/proposal.md +++ b/openspec/changes/export-import-prompts-and-assistants/proposal.md @@ -10,9 +10,9 @@ A prompt belongs to a workspace and an assistant belongs to a deployment, but bo ## What Changes -- `langstar prompt export ` writes one JSON file holding a saved prompt commit exactly as LangSmith returns it with the model included: message templates, input variables, output schema, structured output method, model binding, the repo's description, readme and tags, the commit tags, and where it came from. +- `langstar prompt export ` writes one JSON file holding a saved prompt commit as LangSmith returns it with the model included: message templates, input variables, output schema, structured output method, model binding, the repo's description, readme and tags, the commit tags, and where it came from. - `langstar prompt import --workspace-id ` creates the prompt in the destination workspace, or adds a commit to an existing prompt when the user passes `--update`. `--dry-run` shows what it would do. After writing, it reads the prompt back and reports any difference. -- `langstar prompt compare ` reports the differences between the file and a prompt in a workspace. `langstar prompt verify ` also checks what the prompt needs in that workspace: the secrets its model reads and a model configuration for its provider. +- `langstar prompt compare ` reports the differences between the file and a prompt in a workspace. `langstar prompt verify ` also checks that the workspace has the secrets the prompt's model reads, and notes when the workspace has no saved model configuration for the model's provider. - `langstar assistant export`, `import`, `compare` and `verify` do the same for assistants on a deployment. The file holds the name, description, graph ID, configuration, context, metadata and any prompt names the assistant refers to. `verify` checks that the destination deployment has the graph and that the destination workspace has the referenced prompts. - Import records each destination ID (the new commit hash, or the new assistant ID) in a separate file next to the export, so importing the same file again finds the copy it made earlier instead of creating another one. - `compare`, `verify` and `import` share one exit status scheme and print a JSON report with `-f json`. @@ -21,7 +21,7 @@ A prompt belongs to a workspace and an assistant belongs to a deployment, but bo ### New Capabilities -- `prompt`: the `langstar prompt` command group. This change adds its export, import, compare and verify requirements; the existing prompt subcommands get their requirements when [#728 Redesign prompt create/get/update commands](https://github.com/codekiln/langstar/issues/728) lands. +- `prompt`: the `langstar prompt` command group. This change adds its export, import, compare and verify requirements; the existing prompt subcommands get their requirements when [#728 Implement CRUD command structure (create, get, update) for prompt commands](https://github.com/codekiln/langstar/issues/728) lands. ### Modified Capabilities @@ -29,9 +29,9 @@ A prompt belongs to a workspace and an assistant belongs to a deployment, but bo ## Impact -- SDK: `sdk/src/prompts.rs` gains a commit read that asks for the model and calls for prompt tags. `sdk/src/assistants.rs` gains the `context` field, and `assistant_id` and `if_exists` on create. -- CLI: new subcommands in `cli/src/commands/prompt.rs` and `cli/src/commands/assistant.rs`, and a shared module for the export file, the destination record, comparison and the exit status. -- Tests need a second workspace and a second deployment, and one model provider key for the structured output check. codekiln creates each one; langstar's tests then read them through `fnox.toml` and the CI workflow. +- SDK: `sdk/src/prompts.rs` gains a commit read that asks for the model and calls for prompt tags. `sdk/src/assistants.rs` gains the `context` and `version` fields, `assistant_id` and `if_exists` on create, and a search by graph ID. +- CLI: new subcommands in `cli/src/commands/prompt.rs` and `cli/src/commands/assistant.rs`, and a shared module for the export file, the destination record, comparison and the exit status. `resolve_deployment_url` in `cli/src/deployment_utils.rs` learns to find the Agent Server URL of a deployment built from GitHub, which every `langstar assistant` command needs. +- Tests need a second workspace, a second deployment and one model provider key for the structured output check. codekiln creates the workspace and the key, and langstar's tests read them through `fnox.toml` and the CI workflow; the `TestDeploymentConfig` fixture creates the deployment on the first run and the tests reuse it after that. - Docs: a new usage page on moving prompts and assistants between environments. ## Citations diff --git a/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md b/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md index fface00d1..97d143caf 100644 --- a/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md +++ b/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md @@ -112,7 +112,7 @@ After writing, `langstar assistant import` SHALL read the assistant back from th - **AND** the workspace has no `support-system` prompt with a `prod` tag - **THEN** langstar reports `support-system:prod` as missing and exits with status 2 -### Requirement: Assistant transfer results share the prompt transfer shape +### Requirement: Assistant compare, verify and import use the prompt commands' exit statuses and JSON report `assistant compare`, `assistant verify` and `assistant import` SHALL use the exit statuses and the JSON report shape of `prompt compare`, `prompt verify` and `prompt import`. diff --git a/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md index ae12313fb..1ee1b1211 100644 --- a/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md +++ b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md @@ -44,7 +44,7 @@ Exporting the same commit twice SHALL produce identical files. The file SHALL ho ### Requirement: List the credentials a prompt needs without their values -The export file SHALL list, under `requires.secrets`, the name of every secret the manifest refers to, and under `requires.model_provider`, the model's provider. It SHALL hold no secret values. +The export file SHALL list, under `requires.secrets`, the name of every secret the manifest refers to, and under `requires.model_provider`, the model's provider class name as LangSmith returns it, such as `ChatOpenAI`. It SHALL hold no secret values. #### Scenario: Model reads an API key secret @@ -147,7 +147,7 @@ After writing, `langstar prompt import` SHALL read the destination commit back w ### Requirement: Verify that a prompt can run in a workspace -`langstar prompt verify ` SHALL do what `compare` does and also check that the workspace has every secret in `requires.secrets` and a model configuration for `requires.model_provider`. It SHALL name each missing secret and provider and exit with status 2 when any check fails. +`langstar prompt verify ` SHALL do what `compare` does and also check that the workspace has every secret in `requires.secrets`. It SHALL name each missing secret and exit with status 2 when any secret is missing. It SHALL also look for a saved model configuration whose settings `id` ends in the class name `requires.model_provider` holds, and report a missing one as a note that leaves the exit status unchanged, because the prompt runs with the model its manifest carries. #### Scenario: Secret missing in the destination @@ -155,7 +155,14 @@ After writing, `langstar prompt import` SHALL read the destination commit back w - **AND** the file lists `OPENAI_API_KEY` under `requires.secrets` - **THEN** langstar reports `OPENAI_API_KEY` as missing and exits with status 2 -### Requirement: Report results in one shape with one set of exit statuses +#### Scenario: No saved model configuration for the provider + +- **WHEN** the file's `requires.model_provider` is `ChatOpenAI` +- **AND** no saved model configuration in the destination workspace has a settings `id` ending in `ChatOpenAI` +- **AND** every secret in `requires.secrets` exists and the prompt matches the file +- **THEN** langstar notes that no saved model configuration matches `ChatOpenAI` and exits with status 0 + +### Requirement: Exit with 0 for a match, 2 for a difference and 1 for an error `prompt compare`, `prompt verify` and `prompt import` SHALL exit with status 0 when the prompt matches and every check passes, 2 when they find a difference or a failed check, and 1 on any other error. With `-f json` each SHALL print one JSON object with `status`, `differences` and `checks`. diff --git a/openspec/changes/export-import-prompts-and-assistants/tasks.md b/openspec/changes/export-import-prompts-and-assistants/tasks.md index 9f3cf8893..a5aeb7520 100644 --- a/openspec/changes/export-import-prompts-and-assistants/tasks.md +++ b/openspec/changes/export-import-prompts-and-assistants/tasks.md @@ -9,16 +9,17 @@ Each numbered group is one pull request, in the order listed. ## 2. Test environments -- [ ] 2.1 Add the destination workspace and the model provider key to `fnox.toml`, the CI workflow and `docs/dev/environment-variables.md`, with placeholders only, after codekiln has created the workspace and the key and stored them in 1Password and the repo's CI secrets; verify with `mise run check` and a CI run that reads both +- [ ] 2.1 Add the destination workspace and the model provider key to `fnox.toml`, the CI workflow and `docs/dev/environment-variables.md`, with placeholders only; verify with `mise run check` and a CI run that reads both. This task waits on codekiln to create a second workspace in the test organization and an OpenAI API key, and to store both in 1Password and the repository's CI secrets. The import tests need the workspace as a place to import into. The key lets the check in 7.3 run each imported structured prompt once against a small OpenAI model, which bills that account on every CI run; it is the one test that shows a copy still returns structured data, where every other test compares text. - [ ] 2.2 Add a destination deployment fixture to `sdk/src/test_utils.rs` and `cli/tests/common/fixtures.rs` that returns a deployment with a reachable URL; verify with an integration test that lists assistants on it +- [ ] 2.3 Make `resolve_deployment_url` in `cli/src/deployment_utils.rs` return the URL of a deployment built from GitHub, which has no `source_config.custom_url`; verify by removing `#[ignore]` from `test_assistant_create_basic` in `cli/tests/assistant_command_test.rs` and running it against the test deployment -## 3. SDK reads and writes the fields a transfer needs +## 3. SDK reads and writes the fields export and import need - [ ] 3.1 Add a commit read that sends `include=model` and returns `model_config` and `model_provider` to `sdk/src/prompts.rs`; verify with an httpmock test that asserts the query string and the parsed fields, and an integration test that reads a prompt with a model - [ ] 3.2 Add listing and creating prompt tags to `sdk/src/prompts.rs`; verify with httpmock tests and an integration test that creates a tag and reads it back through the SDK -- [ ] 3.3 Add `context` to `Assistant`, `CreateAssistantRequest` and `UpdateAssistantRequest`, and `assistant_id` and `if_exists` to `CreateAssistantRequest`, in `sdk/src/assistants.rs`; verify with httpmock tests of the request bodies and an integration test that creates an assistant with context and reads it back +- [ ] 3.3 Add `context` and `version` to `Assistant`, `context` to `CreateAssistantRequest` and `UpdateAssistantRequest`, `assistant_id` and `if_exists` to `CreateAssistantRequest`, and a `graph_id` filter to `AssistantSearchRequest`, in `sdk/src/assistants.rs`; verify with httpmock tests of the request bodies and an integration test that creates an assistant with context, reads it back with its version, and finds it by graph ID -## 4. Shared transfer module +## 4. Code the prompt and assistant commands share: the export file, the comparison and the report - [ ] 4.1 Add the export file types, with sorted keys and no export time, and the destination record file, to a new module in `cli/src/`; verify with unit tests that serialize the same value twice to identical text and that updating one destination leaves the others unchanged - [ ] 4.2 Add the JSON comparison that drops server-set fields and reports differences by JSON Pointer; verify with unit tests on nested objects, arrays, missing keys and ignored fields @@ -33,7 +34,7 @@ Each numbered group is one pull request, in the order listed. ## 6. prompt compare and verify - [ ] 6.1 Add `langstar prompt compare`; verify with integration tests that export a prompt, change its model temperature through the SDK, and assert the reported path, both values and exit status 2 -- [ ] 6.2 Add `langstar prompt verify` with the secret and model configuration checks; verify with integration tests against a workspace that lacks a listed secret and one that has it +- [ ] 6.2 Add `langstar prompt verify` with the secret check and the saved model configuration note, matching `requires.model_provider` against the class name at the end of each configuration's settings `id`; verify with integration tests against a workspace that lacks a listed secret and one that has it, and a test that a missing saved model configuration is reported and leaves the exit status at 0 - [ ] 6.3 Document compare and verify; verify that each documented command runs as written ## 7. prompt import @@ -51,7 +52,7 @@ Each numbered group is one pull request, in the order listed. ## 9. assistant compare and verify - [ ] 9.1 Add `langstar assistant compare`; verify with an integration test that changes metadata through the SDK and asserts the reported path and exit status 2 -- [ ] 9.2 Add `langstar assistant verify` with the graph and prompt checks; verify with integration tests for a missing graph ID and a missing prompt tag +- [ ] 9.2 Add `langstar assistant verify` with the graph check and the prompt check, which reads `GET /api/v1/commits/{owner}/{repo}/{tag}`; verify with integration tests for a missing graph ID and a missing prompt tag - [ ] 9.3 Document assistant compare and verify; verify that each documented command runs as written ## 10. assistant import @@ -59,7 +60,7 @@ Each numbered group is one pull request, in the order listed. - [ ] 10.1 Add `langstar assistant import` with creation, read-back and the destination record; verify with an integration test that imports into the destination deployment and checks name, graph ID, configuration, context, metadata and prompt references through the Agent Server API - [ ] 10.2 Add `--update`, `--assistant-id` and `--dry-run`; verify with integration tests that a repeated import creates no assistant, that `--update` changes the recorded one, and that a deleted recorded assistant exits with status 1 - [ ] 10.3 Add an end-to-end test that imports prompts into the destination workspace and then an assistant that refers to them onto the destination deployment, and runs `assistant verify`; verify that it passes and that it fails after deleting one prompt tag -- [ ] 10.4 Document assistant import, the deployment scope of assistants, and how export and import relate to #728's `create`, `get` and `update`; verify that each documented command runs as written +- [ ] 10.4 Document assistant import, the deployment scope of assistants, and how export and import relate to the `create`, `get` and `update` commands of [#728 Implement CRUD command structure (create, get, update) for prompt commands](https://github.com/codekiln/langstar/issues/728); verify that each documented command runs as written ## Workflow follow-up From 7c26daa8619a877bc03cf0a98e01391dcc2e7f84 Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Tue, 13 Oct 2026 18:23:03 -0400 Subject: [PATCH 03/90] =?UTF-8?q?=F0=9F=93=9A=20docs(openspec):=20carry=20?= =?UTF-8?q?the=20commit=20description,=20repo=20details=20and=20metadata?= =?UTF-8?q?=20removals=20through=20import?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Export now writes the commit description, and import sends it on each commit it creates. With --update, prompt import also sets the destination repo's description, readme, tags and visibility through PATCH /api/v1/repos/{owner}/{repo}, and a repo-only difference updates the repo without adding a commit. Assistant import sends each metadata key the destination has and the file lacks with a null value, because the Agent Server merges metadata on update, and reports any key that remains. Task 3.3 adds graph_id to UpdateAssistantRequest, task 3.4 adds the commit description and repo update to the SDK, and task 4.1 tests sorted keys with values built in different insertion orders. #807 Export, import, compare, and verify prompts and assistants across LangSmith environments Co-Authored-By: Claude Opus 5.5 --- .../design.md | 10 ++++--- .../proposal.md | 4 +-- .../specs/assistant/spec.md | 15 ++++++++++- .../specs/prompt/spec.md | 27 ++++++++++++++++--- .../tasks.md | 17 ++++++------ 5 files changed, 55 insertions(+), 18 deletions(-) diff --git a/openspec/changes/export-import-prompts-and-assistants/design.md b/openspec/changes/export-import-prompts-and-assistants/design.md index 0379fbe02..629b506f5 100644 --- a/openspec/changes/export-import-prompts-and-assistants/design.md +++ b/openspec/changes/export-import-prompts-and-assistants/design.md @@ -42,7 +42,7 @@ One prompt export looks like this: "format_version": 1, "source": { "workspace_id": "", "handle": "my-prompt", "commit_hash": "", "ref": "prod" }, "repo": { "handle": "my-prompt", "description": "...", "readme": "...", "tags": ["support"], "is_public": false }, - "commit": { "manifest": { "lc": 1, "type": "constructor", "id": ["langchain", "schema", "runnable", "RunnableSequence"], "kwargs": { "first": {}, "last": {} } }, "model_config": {}, "model_provider": "ChatOpenAI", "tags": ["prod"] }, + "commit": { "description": "...", "manifest": { "lc": 1, "type": "constructor", "id": ["langchain", "schema", "runnable", "RunnableSequence"], "kwargs": { "first": {}, "last": {} } }, "model_config": {}, "model_provider": "ChatOpenAI", "tags": ["prod"] }, "requires": { "secrets": ["OPENAI_API_KEY"], "model_provider": "ChatOpenAI" } } ``` @@ -61,11 +61,15 @@ Import writes the destination's IDs to `.destinations.json`, keyed by ### Import changes an existing resource only with --update -A prompt that exists in the destination gets a new commit only with `--update`, and its parent is the destination's latest commit, so LangSmith keeps the earlier history. An assistant that exists gets updated only with `--update`, which makes the Agent Server save a new version. Without `--update`, import lists the differences and exits with status 2. `--dry-run` reports the action import would take and writes nothing. +A prompt that exists in the destination gets a new commit only with `--update`, and its parent is the destination's latest commit, so LangSmith keeps the earlier history. Import sends the file's commit description with each commit it creates. With `--update`, import also sets the destination repo's description, readme, tags and visibility to the file's values through `PATCH /api/v1/repos/{owner}/{repo}` when they differ. When only those repo fields differ and the manifest matches, import updates the repo and adds no commit. + +An assistant that exists gets updated only with `--update`, which makes the Agent Server save a new version. The Agent Server merges the metadata an update sends into the assistant's metadata (`AssistantPatch` in `reference/openapi/langchain/agent-server/openapi.json`), so sending the file's metadata alone would keep every key the destination has and the file lacks. Import sends each such key with a null value and then reads the assistant back. [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](https://github.com/codekiln/langstar/blob/aeb831e1e048a24a6c2c9ea7761d5131fc3ad2c2/openspec/changes/export-import-prompts-and-assistants/tasks.md#L8) checks against the live server whether a null value removes the key. If it does not, the read-back finds the leftover keys, and import reports them as differences and exits with status 2. Import also sends the file's graph ID, so an assistant whose graph changed at the source moves to that graph. + +Without `--update`, import lists the differences and exits with status 2. `--dry-run` reports the action import would take and writes nothing. ### Compare reports differences by JSON path -Compare reads the destination, removes the fields the server sets (IDs, version numbers, timestamps, owner and like counts), and walks both JSON documents. Each difference names its JSON Pointer path and both values. Import runs the same comparison after it writes, so a field LangSmith or the Agent Server drops shows up as a difference. +Compare reads the destination, removes the fields the server sets (IDs, version numbers, timestamps, owner and like counts), and walks both JSON documents. Each difference names its JSON Pointer path and both values. For a prompt, the comparison covers the commit description and the repo's description, readme, tags and visibility as well as the manifest. Import runs the same comparison after it writes, so a field LangSmith or the Agent Server drops shows up as a difference. ### Verify checks what the copy needs to run diff --git a/openspec/changes/export-import-prompts-and-assistants/proposal.md b/openspec/changes/export-import-prompts-and-assistants/proposal.md index 43f3b4970..784e2c5a9 100644 --- a/openspec/changes/export-import-prompts-and-assistants/proposal.md +++ b/openspec/changes/export-import-prompts-and-assistants/proposal.md @@ -11,7 +11,7 @@ A prompt belongs to a workspace and an assistant belongs to a deployment, but bo ## What Changes - `langstar prompt export ` writes one JSON file holding a saved prompt commit as LangSmith returns it with the model included: message templates, input variables, output schema, structured output method, model binding, the repo's description, readme and tags, the commit tags, and where it came from. -- `langstar prompt import --workspace-id ` creates the prompt in the destination workspace, or adds a commit to an existing prompt when the user passes `--update`. `--dry-run` shows what it would do. After writing, it reads the prompt back and reports any difference. +- `langstar prompt import --workspace-id ` creates the prompt in the destination workspace, or adds a commit to an existing prompt and updates its repo details when the user passes `--update`. `--dry-run` shows what it would do. After writing, it reads the prompt back and reports any difference. - `langstar prompt compare ` reports the differences between the file and a prompt in a workspace. `langstar prompt verify ` also checks that the workspace has the secrets the prompt's model reads, and notes when the workspace has no saved model configuration for the model's provider. - `langstar assistant export`, `import`, `compare` and `verify` do the same for assistants on a deployment. The file holds the name, description, graph ID, configuration, context, metadata and any prompt names the assistant refers to. `verify` checks that the destination deployment has the graph and that the destination workspace has the referenced prompts. - Import records each destination ID (the new commit hash, or the new assistant ID) in a separate file next to the export, so importing the same file again finds the copy it made earlier instead of creating another one. @@ -29,7 +29,7 @@ A prompt belongs to a workspace and an assistant belongs to a deployment, but bo ## Impact -- SDK: `sdk/src/prompts.rs` gains a commit read that asks for the model and calls for prompt tags. `sdk/src/assistants.rs` gains the `context` and `version` fields, `assistant_id` and `if_exists` on create, and a search by graph ID. +- SDK: `sdk/src/prompts.rs` gains a commit read that asks for the model, a commit description on commit create, a repo update and calls for prompt tags. `sdk/src/assistants.rs` gains the `context` and `version` fields, `assistant_id` and `if_exists` on create, `graph_id` on update, and a search by graph ID. - CLI: new subcommands in `cli/src/commands/prompt.rs` and `cli/src/commands/assistant.rs`, and a shared module for the export file, the destination record, comparison and the exit status. `resolve_deployment_url` in `cli/src/deployment_utils.rs` learns to find the Agent Server URL of a deployment built from GitHub, which every `langstar assistant` command needs. - Tests need a second workspace, a second deployment and one model provider key for the structured output check. codekiln creates the workspace and the key, and langstar's tests read them through `fnox.toml` and the CI workflow; the `TestDeploymentConfig` fixture creates the deployment on the first run and the tests reuse it after that. - Docs: a new usage page on moving prompts and assistants between environments. diff --git a/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md b/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md index 97d143caf..ae3580166 100644 --- a/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md +++ b/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md @@ -57,7 +57,7 @@ When the destination record holds an assistant ID for the deployment, `langstar ### Requirement: Update an existing assistant only on request -`langstar assistant import --update` SHALL update the recorded assistant, or the assistant named by `--assistant-id `, with the file's name, description, graph ID, configuration, context and metadata. `--assistant-id` SHALL take precedence over the destination record. +`langstar assistant import --update` SHALL update the recorded assistant, or the assistant named by `--assistant-id `, with the file's name, description, graph ID, configuration, context and metadata. `--assistant-id` SHALL take precedence over the destination record. It SHALL remove each metadata key the destination assistant has and the file lacks. When such a key remains after the update, it SHALL list that key as a difference and exit with status 2. #### Scenario: Update the recorded assistant @@ -70,6 +70,19 @@ When the destination record holds an assistant ID for the deployment, `langstar - **WHEN** the user runs `langstar assistant import support-bot.json --deployment my-agent-prod --update --assistant-id ` - **THEN** langstar updates that assistant and records its ID for `my-agent-prod` +#### Scenario: Remove a metadata key the file lacks + +- **WHEN** the recorded assistant's metadata is `{"kept": 1, "stale": 2}` +- **AND** the file's metadata is `{"kept": 1}` +- **AND** the user runs `langstar assistant import support-bot.json --deployment my-agent-prod --update` +- **THEN** the assistant's metadata read back from the deployment is `{"kept": 1}` +- **AND** langstar exits with status 0 + +#### Scenario: Deployment keeps a removed metadata key + +- **WHEN** the Agent Server keeps `stale` in the assistant's metadata after the update +- **THEN** langstar lists `/metadata/stale` as a difference and exits with status 2 + ### Requirement: Dry run an assistant import `langstar assistant import --dry-run` SHALL report whether it would create the assistant, update it, or leave it unchanged, list the differences, and change nothing on the deployment or in the destination record. diff --git a/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md index 1ee1b1211..dc94a35a7 100644 --- a/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md +++ b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md @@ -6,13 +6,14 @@ ### Requirement: Export a prompt commit to a file -`langstar prompt export ` SHALL write one JSON file holding the prompt's repo details (handle, description, readme, tags, visibility), the commit's manifest as LangSmith returns it with the model included, the commit's model configuration and provider, the commit tags that point at it, and a `source` object naming the workspace, handle and commit hash. `--output ` SHALL choose the file; without it the file goes to stdout. +`langstar prompt export ` SHALL write one JSON file holding the prompt's repo details (handle, description, readme, tags, visibility), the commit's description, the commit's manifest as LangSmith returns it with the model included, the commit's model configuration and provider, the commit tags that point at it, and a `source` object naming the workspace, handle and commit hash. `--output ` SHALL choose the file; without it the file goes to stdout. #### Scenario: Export the latest commit - **WHEN** a user runs `langstar prompt export my-prompt --output my-prompt.json` - **THEN** langstar writes `my-prompt.json` with the manifest of the latest commit, including the model binding - **AND** `source.commit_hash` holds that commit's hash +- **AND** `commit.description` holds that commit's description #### Scenario: Export a structured prompt @@ -63,7 +64,7 @@ The export file SHALL list, under `requires.secrets`, the name of every secret t ### Requirement: Import a prompt that does not exist yet -When the destination workspace has no prompt with the file's handle, `langstar prompt import` SHALL create the prompt with the file's repo details, push the file's manifest as its first commit, and create the file's commit tags on that commit. +When the destination workspace has no prompt with the file's handle, `langstar prompt import` SHALL create the prompt with the file's repo details, push the file's manifest and commit description as its first commit, and create the file's commit tags on that commit. #### Scenario: First import @@ -74,7 +75,7 @@ When the destination workspace has no prompt with the file's handle, `langstar p ### Requirement: Change an existing prompt only with --update -When the destination prompt exists and its latest commit differs from the file, `langstar prompt import` SHALL change nothing and exit with status 2, listing the differences, unless the user passes `--update`. With `--update` it SHALL push the file's manifest as a new commit whose parent is the destination's latest commit, and move the file's tags to it. +When the destination prompt exists and its latest commit differs from the file, `langstar prompt import` SHALL change nothing and exit with status 2, listing the differences, unless the user passes `--update`. With `--update` it SHALL push the file's manifest and commit description as a new commit whose parent is the destination's latest commit, and move the file's tags to it. #### Scenario: Destination differs and --update is missing @@ -87,11 +88,24 @@ When the destination prompt exists and its latest commit differs from the file, - **WHEN** the user runs the same import with `--update` - **THEN** langstar adds a commit to the destination prompt holding the file's manifest + #### Scenario: Destination already matches - **WHEN** the destination prompt's latest commit matches the file - **THEN** langstar pushes nothing, reports that the prompt is unchanged and exits with status 0 +### Requirement: Update the repo details only with --update + +When the destination repo's description, readme, tags or visibility differ from the file, `langstar prompt import` SHALL treat them as differences. With `--update` it SHALL set those fields to the file's values. When only those fields differ and the latest commit matches the file, it SHALL update the repo and add no commit. + +#### Scenario: Only the repo readme differs + +- **WHEN** the destination prompt's latest commit matches the file +- **AND** the destination repo's readme differs from the file's readme +- **AND** the user runs `langstar prompt import my-prompt.json --workspace-id --update` +- **THEN** langstar sets the destination repo's readme to the file's readme and adds no commit +- **AND** exits with status 0 + ### Requirement: Record the destination commit After a successful import, langstar SHALL record the destination workspace, handle and commit hash in a destination record file, `.destinations.json` next to the export unless `--record ` names another. Each import SHALL add or replace only the entry for its own destination workspace. @@ -133,7 +147,7 @@ After writing, `langstar prompt import` SHALL read the destination commit back w ### Requirement: Compare a prompt with its export file -`langstar prompt compare ` SHALL read the prompt with the file's handle from the workspace, with the model included, and list each difference by its JSON path with the file's value and the workspace's value. `--commit ` SHALL choose the commit; without it langstar SHALL read the latest commit. Fields the server sets, such as IDs and timestamps, SHALL be left out of the comparison. +`langstar prompt compare ` SHALL read the prompt with the file's handle from the workspace, with the model included, and its repo details, and list each difference by its JSON path with the file's value and the workspace's value. `--commit ` SHALL choose the commit; without it langstar SHALL read the latest commit. Fields the server sets, such as IDs and timestamps, SHALL be left out of the comparison. #### Scenario: Prompt matches @@ -145,6 +159,11 @@ After writing, `langstar prompt import` SHALL read the destination commit back w - **WHEN** someone changed the model temperature in the workspace's prompt - **THEN** langstar lists the temperature's JSON path with both values and exits with status 2 +#### Scenario: Repo readme changed + +- **WHEN** someone changed the readme of the workspace's prompt +- **THEN** langstar lists the readme's JSON path with both values and exits with status 2 + ### Requirement: Verify that a prompt can run in a workspace `langstar prompt verify ` SHALL do what `compare` does and also check that the workspace has every secret in `requires.secrets`. It SHALL name each missing secret and exit with status 2 when any secret is missing. It SHALL also look for a saved model configuration whose settings `id` ends in the class name `requires.model_provider` holds, and report a missing one as a note that leaves the exit status unchanged, because the prompt runs with the model its manifest carries. diff --git a/openspec/changes/export-import-prompts-and-assistants/tasks.md b/openspec/changes/export-import-prompts-and-assistants/tasks.md index a5aeb7520..039562324 100644 --- a/openspec/changes/export-import-prompts-and-assistants/tasks.md +++ b/openspec/changes/export-import-prompts-and-assistants/tasks.md @@ -5,7 +5,7 @@ Each numbered group is one pull request, in the order listed. ## 1. Check the API round trip before building on it - [ ] 1.1 Extract the commit get, commit create, repo tags, workspace secrets and playground settings endpoints from `reference/openapi/langchain/langsmith/openapi.json` into `reference/api-specs/langsmith/` and add their jq queries to `FRAGMENTS.md`; verify by rerunning each query and getting the same file -- [ ] 1.2 Write `reference/research/807-openapi-validation.md`, recording from the live API: the manifest returned with `include=model` for a structured prompt with a model, whether pushing that manifest to a second workspace returns the same model, what pushing an identical manifest returns, what the Agent Server returns for `context`, and what `POST /api/v1/prompts/invoke_prompt` returns for a prompt with a model; verify that every claim in the report names the request that produced it +- [ ] 1.2 Write `reference/research/807-openapi-validation.md`, recording from the live API: the manifest returned with `include=model` for a structured prompt with a model, whether pushing that manifest to a second workspace returns the same model, what pushing an identical manifest returns, what the Agent Server returns for `context`, whether an Agent Server assistant update that sends a metadata key with a null value removes that key, and what `POST /api/v1/prompts/invoke_prompt` returns for a prompt with a model; verify that every claim in the report names the request that produced it ## 2. Test environments @@ -15,13 +15,14 @@ Each numbered group is one pull request, in the order listed. ## 3. SDK reads and writes the fields export and import need -- [ ] 3.1 Add a commit read that sends `include=model` and returns `model_config` and `model_provider` to `sdk/src/prompts.rs`; verify with an httpmock test that asserts the query string and the parsed fields, and an integration test that reads a prompt with a model +- [ ] 3.1 Add a commit read that sends `include=model` and returns `description`, `model_config` and `model_provider` to `sdk/src/prompts.rs`; verify with an httpmock test that asserts the query string and the parsed fields, and an integration test that reads a prompt with a model - [ ] 3.2 Add listing and creating prompt tags to `sdk/src/prompts.rs`; verify with httpmock tests and an integration test that creates a tag and reads it back through the SDK -- [ ] 3.3 Add `context` and `version` to `Assistant`, `context` to `CreateAssistantRequest` and `UpdateAssistantRequest`, `assistant_id` and `if_exists` to `CreateAssistantRequest`, and a `graph_id` filter to `AssistantSearchRequest`, in `sdk/src/assistants.rs`; verify with httpmock tests of the request bodies and an integration test that creates an assistant with context, reads it back with its version, and finds it by graph ID +- [ ] 3.3 Add `context` and `version` to `Assistant`, `context` and `graph_id` to `UpdateAssistantRequest`, `context` to `CreateAssistantRequest`, `assistant_id` and `if_exists` to `CreateAssistantRequest`, and a `graph_id` filter to `AssistantSearchRequest`, in `sdk/src/assistants.rs`; verify with httpmock tests of the request bodies and an integration test that creates an assistant with context, reads it back with its version, finds it by graph ID, and updates its graph ID and reads the new graph ID back +- [ ] 3.4 Add `description` to `CommitRequest`, and a repo update that sends `PATCH /api/v1/repos/{owner}/{repo}` with `description`, `readme`, `tags` and `is_public`, to `sdk/src/prompts.rs`; verify with httpmock tests of the request bodies and an integration test that pushes a commit with a description, updates the repo's description, readme, tags and visibility, and reads each back ## 4. Code the prompt and assistant commands share: the export file, the comparison and the report -- [ ] 4.1 Add the export file types, with sorted keys and no export time, and the destination record file, to a new module in `cli/src/`; verify with unit tests that serialize the same value twice to identical text and that updating one destination leaves the others unchanged +- [ ] 4.1 Add the export file types, with sorted keys and no export time, and the destination record file, to a new module in `cli/src/`; verify with unit tests that build equivalent nested values with keys inserted in different orders, serialize each, and assert identical text with keys sorted at every level, and that updating one destination leaves the others unchanged - [ ] 4.2 Add the JSON comparison that drops server-set fields and reports differences by JSON Pointer; verify with unit tests on nested objects, arrays, missing keys and ignored fields - [ ] 4.3 Add the shared result report and the exit statuses 0, 2 and 1, printing JSON to stdout and progress to stderr; verify with unit tests of the report and a CLI test that checks the exit status - [ ] 4.4 Add the secret reference walk that lists secret names from a manifest; verify with unit tests on the manifest fixtures from `references/langsmith-sdk.md` @@ -33,14 +34,14 @@ Each numbered group is one pull request, in the order listed. ## 6. prompt compare and verify -- [ ] 6.1 Add `langstar prompt compare`; verify with integration tests that export a prompt, change its model temperature through the SDK, and assert the reported path, both values and exit status 2 +- [ ] 6.1 Add `langstar prompt compare`; verify with integration tests that export a prompt, change its model temperature through the SDK, and assert the reported path, both values and exit status 2, and that a changed commit description and a changed repo readme are each reported the same way - [ ] 6.2 Add `langstar prompt verify` with the secret check and the saved model configuration note, matching `requires.model_provider` against the class name at the end of each configuration's settings `id`; verify with integration tests against a workspace that lacks a listed secret and one that has it, and a test that a missing saved model configuration is reported and leaves the exit status at 0 - [ ] 6.3 Document compare and verify; verify that each documented command runs as written ## 7. prompt import -- [ ] 7.1 Add `langstar prompt import` with required `--workspace-id`, creating the repo, the commit and its tags, then reading the commit back; verify with integration tests that export from the test workspace, import into the destination workspace, and compare the full source and destination commits through the SDK with the model included, for every prompt shape from 5.1 -- [ ] 7.2 Add `--update`, `--dry-run`, `--record` and the destination record; verify with integration tests that a repeated import creates no commit, that a changed destination is refused without `--update`, that `--update` adds one commit with the right parent, and that `--dry-run` writes nothing +- [ ] 7.1 Add `langstar prompt import` with required `--workspace-id`, creating the repo, the commit and its tags, then reading the commit back; verify with integration tests that export from the test workspace, import into the destination workspace, and compare the full source and destination commits through the SDK with the model included, the commit description among them, and the repo's description, readme, tags and visibility, for every prompt shape from 5.1 +- [ ] 7.2 Add `--update`, `--dry-run`, `--record` and the destination record; verify with integration tests that a repeated import creates no commit, that a changed destination is refused without `--update`, that `--update` adds one commit with the right parent and the file's commit description, that `--update` against a destination whose repo description, readme, tags or visibility differ while the manifest matches updates the repo and adds no commit, and that `--dry-run` writes nothing - [ ] 7.3 Add the structured output check as a mise task that pulls the imported prompt with `include_model=True`, invokes it and validates the result against the schema; verify by running it from the import integration test for both methods - [ ] 7.4 Document prompt import, the workspace scope of prompts and setting up destination credentials; verify that each documented command runs as written @@ -58,7 +59,7 @@ Each numbered group is one pull request, in the order listed. ## 10. assistant import - [ ] 10.1 Add `langstar assistant import` with creation, read-back and the destination record; verify with an integration test that imports into the destination deployment and checks name, graph ID, configuration, context, metadata and prompt references through the Agent Server API -- [ ] 10.2 Add `--update`, `--assistant-id` and `--dry-run`; verify with integration tests that a repeated import creates no assistant, that `--update` changes the recorded one, and that a deleted recorded assistant exits with status 1 +- [ ] 10.2 Add `--update`, `--assistant-id` and `--dry-run`; verify with integration tests that a repeated import creates no assistant, that `--update` changes the recorded one, its graph ID among the fields, that `--update` of an assistant with metadata `{"kept": 1, "stale": 2}` from a file with `{"kept": 1}` leaves `{"kept": 1}` or reports `/metadata/stale` and exits with status 2, whichever task 1.2 found the server does, and that a deleted recorded assistant exits with status 1 - [ ] 10.3 Add an end-to-end test that imports prompts into the destination workspace and then an assistant that refers to them onto the destination deployment, and runs `assistant verify`; verify that it passes and that it fails after deleting one prompt tag - [ ] 10.4 Document assistant import, the deployment scope of assistants, and how export and import relate to the `create`, `get` and `update` commands of [#728 Implement CRUD command structure (create, get, update) for prompt commands](https://github.com/codekiln/langstar/issues/728); verify that each documented command runs as written From 4e053624b031d2d8ea395bac9d7b11fa8840ab5d Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Tue, 13 Oct 2026 18:32:54 -0400 Subject: [PATCH 04/90] =?UTF-8?q?=F0=9F=93=9A=20docs(openspec):=20add=20ta?= =?UTF-8?q?g=20moves=20and=20the=20change=20warning=20to=20the=20tasks,=20?= =?UTF-8?q?and=20order=20assistant=20compare=20before=20its=20record?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Task 3.2 now moves a tag with PATCH /api/v1/repos/{owner}/{repo}/tags/{tag_name}, task 1.1 extracts that endpoint and the repo update, task 7.2 tests the warning that the destination changed since the last import, and tasks 9.1 and 9.2 add --assistant-id so assistant compare and verify can name an assistant before import exists. The manifest example uses the PromptPlayground id the upstream fixture holds, and design.md links tasks.md relatively. #807 Export, import, compare, and verify prompts and assistants across LangSmith environments Co-Authored-By: Claude Opus 5.5 --- .../design.md | 16 ++++++++-------- .../proposal.md | 4 ++-- .../references/langsmith-sdk.md | 2 +- .../specs/assistant/spec.md | 2 +- .../tasks.md | 18 +++++++++--------- 5 files changed, 21 insertions(+), 21 deletions(-) diff --git a/openspec/changes/export-import-prompts-and-assistants/design.md b/openspec/changes/export-import-prompts-and-assistants/design.md index 629b506f5..06299a572 100644 --- a/openspec/changes/export-import-prompts-and-assistants/design.md +++ b/openspec/changes/export-import-prompts-and-assistants/design.md @@ -24,7 +24,7 @@ Langstar reads one API key and one workspace at a time, from flags, environment ## Decisions -### Export writes a file with the source's credentials, and import reads it with the destination's +### Export and import run at different times, each with its own environment's API key Export runs with the source environment's credentials and writes a file. Import runs later with the destination's credentials and reads it. Each run uses langstar's existing settings, so a user moving a prompt to another organization sets a different `LANGSMITH_API_KEY` for the import. Langstar needs no second set of credentials and no profiles. The file sits in Git between the two runs, so a reviewer sees the prompt or assistant change in a pull request. @@ -42,7 +42,7 @@ One prompt export looks like this: "format_version": 1, "source": { "workspace_id": "", "handle": "my-prompt", "commit_hash": "", "ref": "prod" }, "repo": { "handle": "my-prompt", "description": "...", "readme": "...", "tags": ["support"], "is_public": false }, - "commit": { "description": "...", "manifest": { "lc": 1, "type": "constructor", "id": ["langchain", "schema", "runnable", "RunnableSequence"], "kwargs": { "first": {}, "last": {} } }, "model_config": {}, "model_provider": "ChatOpenAI", "tags": ["prod"] }, + "commit": { "description": "...", "manifest": { "lc": 1, "type": "constructor", "id": ["langsmith", "playground", "PromptPlayground"], "kwargs": { "first": {}, "last": {} } }, "model_config": {}, "model_provider": "ChatOpenAI", "tags": ["prod"] }, "requires": { "secrets": ["OPENAI_API_KEY"], "model_provider": "ChatOpenAI" } } ``` @@ -53,7 +53,7 @@ The file holds no export time and langstar sorts its keys, so exporting an uncha ### Export asks LangSmith for the model -Export reads the commit with `include=model`, the form the Go SDK uses (`references/langsmith-go.md`). The Python SDK sends the older `include_model=true`, and the LangSmith OpenAPI spec accepts both. With the model included, the manifest becomes a `RunnableSequence` whose last step binds the chat model and its settings, and the response also carries `model_config` and `model_provider` beside the manifest (`references/langsmith-sdk.md`). No field of the create-commit request holds the model, so the model reaches the destination inside the manifest. [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](https://github.com/codekiln/langstar/blob/aeb831e1e048a24a6c2c9ea7761d5131fc3ad2c2/openspec/changes/export-import-prompts-and-assistants/tasks.md#L8) pushes that manifest to a second workspace and records whether LangSmith returns the same model. +Export reads the commit with `include=model`, the form the Go SDK uses (`references/langsmith-go.md`). The Python SDK sends the older `include_model=true`, and the LangSmith OpenAPI spec accepts both. With the model included, the manifest wraps the prompt and the chat model with its settings in one constructor, and the response also carries `model_config` and `model_provider` beside the manifest (`references/langsmith-sdk.md`). No field of the create-commit request holds the model, so the model reaches the destination inside the manifest. [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8) pushes that manifest to a second workspace, records whether LangSmith returns the same model, and records which outer `id` the live API returns. ### Import records what it created in a separate file @@ -63,7 +63,7 @@ Import writes the destination's IDs to `.destinations.json`, keyed by A prompt that exists in the destination gets a new commit only with `--update`, and its parent is the destination's latest commit, so LangSmith keeps the earlier history. Import sends the file's commit description with each commit it creates. With `--update`, import also sets the destination repo's description, readme, tags and visibility to the file's values through `PATCH /api/v1/repos/{owner}/{repo}` when they differ. When only those repo fields differ and the manifest matches, import updates the repo and adds no commit. -An assistant that exists gets updated only with `--update`, which makes the Agent Server save a new version. The Agent Server merges the metadata an update sends into the assistant's metadata (`AssistantPatch` in `reference/openapi/langchain/agent-server/openapi.json`), so sending the file's metadata alone would keep every key the destination has and the file lacks. Import sends each such key with a null value and then reads the assistant back. [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](https://github.com/codekiln/langstar/blob/aeb831e1e048a24a6c2c9ea7761d5131fc3ad2c2/openspec/changes/export-import-prompts-and-assistants/tasks.md#L8) checks against the live server whether a null value removes the key. If it does not, the read-back finds the leftover keys, and import reports them as differences and exits with status 2. Import also sends the file's graph ID, so an assistant whose graph changed at the source moves to that graph. +An assistant that exists gets updated only with `--update`, which makes the Agent Server save a new version. The Agent Server merges the metadata an update sends into the assistant's metadata (`AssistantPatch` in `reference/openapi/langchain/agent-server/openapi.json`), so sending the file's metadata alone would keep every key the destination has and the file lacks. Import sends each such key with a null value and then reads the assistant back. [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8) checks against the live server whether a null value removes the key. If it does not, the read-back finds the leftover keys, and import reports them as differences and exits with status 2. Import also sends the file's graph ID, so an assistant whose graph changed at the source moves to that graph. Without `--update`, import lists the differences and exits with status 2. `--dry-run` reports the action import would take and writes nothing. @@ -87,7 +87,7 @@ Status 0 means the resource matches and every check passed. Status 2 means langs ### A Python script checks that an imported structured prompt still returns structured data -A mise task runs a short Python script that pulls the imported prompt with `include_model=True`, invokes it, and checks that the result matches the prompt's schema. The script exists because the LangSmith API has no documented way to run a prompt with its model: the one prompt endpoint in the OpenAPI spec, `POST /api/v1/prompts/invoke_prompt`, takes message templates and inputs with no model or schema, and no LangSmith SDK calls it. [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](https://github.com/codekiln/langstar/blob/aeb831e1e048a24a6c2c9ea7761d5131fc3ad2c2/openspec/changes/export-import-prompts-and-assistants/tasks.md#L8) tries that endpoint against the live API in case it does more than the spec says. The script needs a model provider API key, which codekiln has to create and add to fnox and CI before the test can run. +A mise task runs a short Python script that pulls the imported prompt with `include_model=True`, invokes it, and checks that the result matches the prompt's schema. The script exists because the LangSmith API has no documented way to run a prompt with its model: the one prompt endpoint in the OpenAPI spec, `POST /api/v1/prompts/invoke_prompt`, takes message templates and inputs with no model or schema, and no LangSmith SDK calls it. [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8) tries that endpoint against the live API in case it does more than the spec says. The script needs a model provider API key, which codekiln has to create and add to fnox and CI before the test can run. ### Export and import use the same SDK calls as the planned `prompt create`, `get` and `update` @@ -95,9 +95,9 @@ A mise task runs a short Python script that pulls the imported prompt with `incl ## Risks / Trade-offs -- [LangSmith may store a pushed model differently from the source] → Import reads the prompt back and reports any difference, and [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](https://github.com/codekiln/langstar/blob/aeb831e1e048a24a6c2c9ea7761d5131fc3ad2c2/openspec/changes/export-import-prompts-and-assistants/tasks.md#L8) checks the round trip before the commands are built. +- [LangSmith may store a pushed model differently from the source] → Import reads the prompt back and reports any difference, and [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8) checks the round trip before the commands are built. - [Pushing a manifest identical to the latest commit may return 409] → Import compares first and pushes nothing when the prompt matches. -- [langstar finds a deployment's Agent Server URL only in `source_config.custom_url` (`resolve_deployment_url` in `cli/src/deployment_utils.rs`), and a deployment built from GitHub has none, so every `langstar assistant` command, the new ones included, fails on such a deployment] → Task 2.3 changes `resolve_deployment_url` to find the URL of a GitHub-built deployment, before any assistant work in groups 8 to 10 starts. +- [langstar finds a deployment's Agent Server URL only in `source_config.custom_url` (`resolve_deployment_url` in `cli/src/deployment_utils.rs`), and a deployment built from GitHub has none, so every `langstar assistant` command, the new ones included, fails on such a deployment] → [export-import-prompts-and-assistants Task 2.3 Return the URL of a deployment built from GitHub](tasks.md#L14) changes `resolve_deployment_url` before anyone builds the assistant export, compare, verify and import commands. - [A tag in the destination may already point at another commit] → `--update` moves the tag, and `--dry-run` lists the move. ## Resolved Questions @@ -123,7 +123,7 @@ None. ### 3 - How should the end-to-end test show that an imported structured prompt still returns structured data? > Context from the drafting agent, for question 3. -> The LangSmith OpenAPI spec's only prompt-invoke endpoint, `POST /api/v1/prompts/invoke_prompt`, takes message templates and inputs with no model or schema, and no SDK calls it. [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](https://github.com/codekiln/langstar/blob/aeb831e1e048a24a6c2c9ea7761d5131fc3ad2c2/openspec/changes/export-import-prompts-and-assistants/tasks.md#L8) tries it live. The draft runs a Python script through a mise task, using the LangSmith Python SDK and LangChain, which is how most users run a prompt. It needs a model provider key, such as `OPENAI_API_KEY`, which codekiln would create and add to fnox and CI. Building the model call into langstar as `prompt verify --invoke` would need langstar to call each provider's API itself. Recommendation: the Python script, with an OpenAI key and a small model. +> The LangSmith OpenAPI spec's only prompt-invoke endpoint, `POST /api/v1/prompts/invoke_prompt`, takes message templates and inputs with no model or schema, and no SDK calls it. [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8) tries it live. The draft runs a Python script through a mise task, using the LangSmith Python SDK and LangChain, which is how most users run a prompt. It needs a model provider key, such as `OPENAI_API_KEY`, which codekiln would create and add to fnox and CI. Building the model call into langstar as `prompt verify --invoke` would need langstar to call each provider's API itself. Recommendation: the Python script, with an OpenAI key and a small model. diff --git a/openspec/changes/export-import-prompts-and-assistants/proposal.md b/openspec/changes/export-import-prompts-and-assistants/proposal.md index 784e2c5a9..c58752f7a 100644 --- a/openspec/changes/export-import-prompts-and-assistants/proposal.md +++ b/openspec/changes/export-import-prompts-and-assistants/proposal.md @@ -10,7 +10,7 @@ A prompt belongs to a workspace and an assistant belongs to a deployment, but bo ## What Changes -- `langstar prompt export ` writes one JSON file holding a saved prompt commit as LangSmith returns it with the model included: message templates, input variables, output schema, structured output method, model binding, the repo's description, readme and tags, the commit tags, and where it came from. +- `langstar prompt export ` writes one JSON file holding a saved prompt commit as LangSmith returns it with the model included: message templates, input variables, output schema, structured output method, model binding and commit description, plus the repo's description, readme, tags and visibility, the commit tags, and where it came from. - `langstar prompt import --workspace-id ` creates the prompt in the destination workspace, or adds a commit to an existing prompt and updates its repo details when the user passes `--update`. `--dry-run` shows what it would do. After writing, it reads the prompt back and reports any difference. - `langstar prompt compare ` reports the differences between the file and a prompt in a workspace. `langstar prompt verify ` also checks that the workspace has the secrets the prompt's model reads, and notes when the workspace has no saved model configuration for the model's provider. - `langstar assistant export`, `import`, `compare` and `verify` do the same for assistants on a deployment. The file holds the name, description, graph ID, configuration, context, metadata and any prompt names the assistant refers to. `verify` checks that the destination deployment has the graph and that the destination workspace has the referenced prompts. @@ -30,7 +30,7 @@ A prompt belongs to a workspace and an assistant belongs to a deployment, but bo ## Impact - SDK: `sdk/src/prompts.rs` gains a commit read that asks for the model, a commit description on commit create, a repo update and calls for prompt tags. `sdk/src/assistants.rs` gains the `context` and `version` fields, `assistant_id` and `if_exists` on create, `graph_id` on update, and a search by graph ID. -- CLI: new subcommands in `cli/src/commands/prompt.rs` and `cli/src/commands/assistant.rs`, and a shared module for the export file, the destination record, comparison and the exit status. `resolve_deployment_url` in `cli/src/deployment_utils.rs` learns to find the Agent Server URL of a deployment built from GitHub, which every `langstar assistant` command needs. +- CLI: new subcommands in `cli/src/commands/prompt.rs` and `cli/src/commands/assistant.rs`, and a shared module for the export file, the destination record, comparison and the exit status. `resolve_deployment_url` in `cli/src/deployment_utils.rs` also returns the Agent Server URL of a deployment built from GitHub, which every `langstar assistant` command needs. - Tests need a second workspace, a second deployment and one model provider key for the structured output check. codekiln creates the workspace and the key, and langstar's tests read them through `fnox.toml` and the CI workflow; the `TestDeploymentConfig` fixture creates the deployment on the first run and the tests reuse it after that. - Docs: a new usage page on moving prompts and assistants between environments. diff --git a/openspec/changes/export-import-prompts-and-assistants/references/langsmith-sdk.md b/openspec/changes/export-import-prompts-and-assistants/references/langsmith-sdk.md index c4b6202be..a62304c21 100644 --- a/openspec/changes/export-import-prompts-and-assistants/references/langsmith-sdk.md +++ b/openspec/changes/export-import-prompts-and-assistants/references/langsmith-sdk.md @@ -4,7 +4,7 @@ Read at [langsmith-sdk 47c553ca](https://github.com/langchain-ai/langsmith-sdk/t [`pull_prompt_commit`](https://github.com/langchain-ai/langsmith-sdk/blob/47c553caf08c1989e2be3ff1886da3eb801f709f/python/langsmith/client.py#L10107) adds `?include_model=true` when asked. The result, [`PromptCommit`](https://github.com/langchain-ai/langsmith-sdk/blob/47c553caf08c1989e2be3ff1886da3eb801f709f/python/langsmith/schemas.py#L1035), holds the manifest plus `model_config` and `model_provider`. -With the model included, the manifest is a `RunnableSequence`. Its `first` step is the `ChatPromptTemplate` or `StructuredPrompt` (with `schema_`), and its `last` step is a `RunnableBinding` around the chat model constructor and its settings. The [integration fixture `prompt_with_model`](https://github.com/langchain-ai/langsmith-sdk/blob/47c553caf08c1989e2be3ff1886da3eb801f709f/python/tests/integration_tests/test_prompts.py#L48) shows one. +With the model included, the manifest in the [integration fixture `prompt_with_model`](https://github.com/langchain-ai/langsmith-sdk/blob/47c553caf08c1989e2be3ff1886da3eb801f709f/python/tests/integration_tests/test_prompts.py#L48) is a constructor with the `id` `["langsmith", "playground", "PromptPlayground"]`, which LangChain loads as a `RunnableSequence`. Its `first` step is the `ChatPromptTemplate` or `StructuredPrompt` (with `schema_`), and its `last` step is a `RunnableBinding` around the chat model constructor and its settings. The chat model refers to its API key as a secret reference, `{"lc": 1, "type": "secret", "id": ["OPENAI_API_KEY"]}`. [`_manifest_has_secrets`](https://github.com/langchain-ai/langsmith-sdk/blob/47c553caf08c1989e2be3ff1886da3eb801f709f/python/langsmith/client.py#L434) finds these by matching objects whose keys are exactly `lc`, `type` and `id`. diff --git a/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md b/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md index ae3580166..124789538 100644 --- a/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md +++ b/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md @@ -39,7 +39,7 @@ - **WHEN** a user runs `langstar assistant import support-bot.json --deployment my-agent-prod` - **AND** the destination record has no entry for `my-agent-prod` - **THEN** langstar creates the assistant and prints its new ID -- **AND** `support-bot.destinations.json` holds `my-agent-prod` with that ID +- **AND** `support-bot.destinations.json` holds the ID of the `my-agent-prod` deployment, with the new assistant ID under it ### Requirement: Find the earlier copy instead of creating another diff --git a/openspec/changes/export-import-prompts-and-assistants/tasks.md b/openspec/changes/export-import-prompts-and-assistants/tasks.md index 039562324..fe7d051d2 100644 --- a/openspec/changes/export-import-prompts-and-assistants/tasks.md +++ b/openspec/changes/export-import-prompts-and-assistants/tasks.md @@ -4,8 +4,8 @@ Each numbered group is one pull request, in the order listed. ## 1. Check the API round trip before building on it -- [ ] 1.1 Extract the commit get, commit create, repo tags, workspace secrets and playground settings endpoints from `reference/openapi/langchain/langsmith/openapi.json` into `reference/api-specs/langsmith/` and add their jq queries to `FRAGMENTS.md`; verify by rerunning each query and getting the same file -- [ ] 1.2 Write `reference/research/807-openapi-validation.md`, recording from the live API: the manifest returned with `include=model` for a structured prompt with a model, whether pushing that manifest to a second workspace returns the same model, what pushing an identical manifest returns, what the Agent Server returns for `context`, whether an Agent Server assistant update that sends a metadata key with a null value removes that key, and what `POST /api/v1/prompts/invoke_prompt` returns for a prompt with a model; verify that every claim in the report names the request that produced it +- [ ] 1.1 Extract the commit get, commit create, repo update (`PATCH /api/v1/repos/{owner}/{repo}`), repo tags (`/api/v1/repos/{owner}/{repo}/tags` and `/api/v1/repos/{owner}/{repo}/tags/{tag_name}`), workspace secrets and playground settings endpoints from `reference/openapi/langchain/langsmith/openapi.json` into `reference/api-specs/langsmith/` and add their jq queries to `FRAGMENTS.md`; verify by rerunning each query and getting the same file +- [ ] 1.2 Write `reference/research/807-openapi-validation.md`, recording from the live API: the manifest returned with `include=model` for a structured prompt with a model and the outer `id` of that manifest, whether pushing that manifest to a second workspace returns the same model, what pushing an identical manifest returns, what the Agent Server returns for `context`, whether an Agent Server assistant update that sends a metadata key with a null value removes that key, and what `POST /api/v1/prompts/invoke_prompt` returns for a prompt with a model; verify that every claim in the report names the request that produced it ## 2. Test environments @@ -16,16 +16,16 @@ Each numbered group is one pull request, in the order listed. ## 3. SDK reads and writes the fields export and import need - [ ] 3.1 Add a commit read that sends `include=model` and returns `description`, `model_config` and `model_provider` to `sdk/src/prompts.rs`; verify with an httpmock test that asserts the query string and the parsed fields, and an integration test that reads a prompt with a model -- [ ] 3.2 Add listing and creating prompt tags to `sdk/src/prompts.rs`; verify with httpmock tests and an integration test that creates a tag and reads it back through the SDK +- [ ] 3.2 Add listing, creating and moving prompt tags to `sdk/src/prompts.rs`, moving a tag with `PATCH /api/v1/repos/{owner}/{repo}/tags/{tag_name}`; verify with httpmock tests and an integration test that creates a tag on one commit, moves it to a second commit, and reads it back through the SDK - [ ] 3.3 Add `context` and `version` to `Assistant`, `context` and `graph_id` to `UpdateAssistantRequest`, `context` to `CreateAssistantRequest`, `assistant_id` and `if_exists` to `CreateAssistantRequest`, and a `graph_id` filter to `AssistantSearchRequest`, in `sdk/src/assistants.rs`; verify with httpmock tests of the request bodies and an integration test that creates an assistant with context, reads it back with its version, finds it by graph ID, and updates its graph ID and reads the new graph ID back - [ ] 3.4 Add `description` to `CommitRequest`, and a repo update that sends `PATCH /api/v1/repos/{owner}/{repo}` with `description`, `readme`, `tags` and `is_public`, to `sdk/src/prompts.rs`; verify with httpmock tests of the request bodies and an integration test that pushes a commit with a description, updates the repo's description, readme, tags and visibility, and reads each back -## 4. Code the prompt and assistant commands share: the export file, the comparison and the report +## 4. Shared code for the prompt and assistant commands: the export file, the comparison and the report - [ ] 4.1 Add the export file types, with sorted keys and no export time, and the destination record file, to a new module in `cli/src/`; verify with unit tests that build equivalent nested values with keys inserted in different orders, serialize each, and assert identical text with keys sorted at every level, and that updating one destination leaves the others unchanged - [ ] 4.2 Add the JSON comparison that drops server-set fields and reports differences by JSON Pointer; verify with unit tests on nested objects, arrays, missing keys and ignored fields - [ ] 4.3 Add the shared result report and the exit statuses 0, 2 and 1, printing JSON to stdout and progress to stderr; verify with unit tests of the report and a CLI test that checks the exit status -- [ ] 4.4 Add the secret reference walk that lists secret names from a manifest; verify with unit tests on the manifest fixtures from `references/langsmith-sdk.md` +- [ ] 4.4 Add the secret reference walk that lists secret names from a manifest; verify with unit tests on a copy of the `prompt_with_model` manifest from the LangSmith Python SDK's integration tests, which `references/langsmith-sdk.md` links, and on a manifest with no secret reference ## 5. prompt export @@ -41,7 +41,7 @@ Each numbered group is one pull request, in the order listed. ## 7. prompt import - [ ] 7.1 Add `langstar prompt import` with required `--workspace-id`, creating the repo, the commit and its tags, then reading the commit back; verify with integration tests that export from the test workspace, import into the destination workspace, and compare the full source and destination commits through the SDK with the model included, the commit description among them, and the repo's description, readme, tags and visibility, for every prompt shape from 5.1 -- [ ] 7.2 Add `--update`, `--dry-run`, `--record` and the destination record; verify with integration tests that a repeated import creates no commit, that a changed destination is refused without `--update`, that `--update` adds one commit with the right parent and the file's commit description, that `--update` against a destination whose repo description, readme, tags or visibility differ while the manifest matches updates the repo and adds no commit, and that `--dry-run` writes nothing +- [ ] 7.2 Add `--update`, `--dry-run`, `--record` and the destination record; verify with integration tests that a repeated import creates no commit, that a changed destination is refused without `--update`, that an import after someone pushed a commit to the destination since the recorded import says the destination changed since the last import and exits with status 2, that `--update` adds one commit with the right parent and the file's commit description, that `--update` against a destination whose repo description, readme, tags or visibility differ while the manifest matches updates the repo and adds no commit, and that `--dry-run` writes nothing - [ ] 7.3 Add the structured output check as a mise task that pulls the imported prompt with `include_model=True`, invokes it and validates the result against the schema; verify by running it from the import integration test for both methods - [ ] 7.4 Document prompt import, the workspace scope of prompts and setting up destination credentials; verify that each documented command runs as written @@ -52,13 +52,13 @@ Each numbered group is one pull request, in the order listed. ## 9. assistant compare and verify -- [ ] 9.1 Add `langstar assistant compare`; verify with an integration test that changes metadata through the SDK and asserts the reported path and exit status 2 -- [ ] 9.2 Add `langstar assistant verify` with the graph check and the prompt check, which reads `GET /api/v1/commits/{owner}/{repo}/{tag}`; verify with integration tests for a missing graph ID and a missing prompt tag +- [ ] 9.1 Add `langstar assistant compare` with `--assistant-id`, and the lookup of the recorded assistant in the destination record from 4.1; verify with an integration test that exports an assistant, changes its metadata through the SDK, runs compare with `--assistant-id` set to that assistant, and asserts the reported path and exit status 2, and a unit test that compare reads the assistant ID from a destination record written by the 4.1 types +- [ ] 9.2 Add `langstar assistant verify` with `--assistant-id`, the graph check and the prompt check, which reads `GET /api/v1/commits/{owner}/{repo}/{tag}`; verify with integration tests that run verify with `--assistant-id` against an exported assistant, one with a file whose graph ID the deployment lacks and one with a missing prompt tag - [ ] 9.3 Document assistant compare and verify; verify that each documented command runs as written ## 10. assistant import -- [ ] 10.1 Add `langstar assistant import` with creation, read-back and the destination record; verify with an integration test that imports into the destination deployment and checks name, graph ID, configuration, context, metadata and prompt references through the Agent Server API +- [ ] 10.1 Add `langstar assistant import` with creation, read-back, the destination record and `--record`; verify with an integration test that imports into the destination deployment and checks name, graph ID, configuration, context, metadata and prompt references through the Agent Server API, and that `--record ` writes the destination record to that path - [ ] 10.2 Add `--update`, `--assistant-id` and `--dry-run`; verify with integration tests that a repeated import creates no assistant, that `--update` changes the recorded one, its graph ID among the fields, that `--update` of an assistant with metadata `{"kept": 1, "stale": 2}` from a file with `{"kept": 1}` leaves `{"kept": 1}` or reports `/metadata/stale` and exits with status 2, whichever task 1.2 found the server does, and that a deleted recorded assistant exits with status 1 - [ ] 10.3 Add an end-to-end test that imports prompts into the destination workspace and then an assistant that refers to them onto the destination deployment, and runs `assistant verify`; verify that it passes and that it fails after deleting one prompt tag - [ ] 10.4 Document assistant import, the deployment scope of assistants, and how export and import relate to the `create`, `get` and `update` commands of [#728 Implement CRUD command structure (create, get, update) for prompt commands](https://github.com/codekiln/langstar/issues/728); verify that each documented command runs as written From 386610016fd48d1ef0dc0dc5f75488d83f196a18 Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Tue, 13 Oct 2026 18:37:42 -0400 Subject: [PATCH 05/90] =?UTF-8?q?=F0=9F=93=9A=20docs(openspec):=20resolve?= =?UTF-8?q?=20an=20assistant's=20deployment=20through=20the=20deployment's?= =?UTF-8?q?=20url?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The assistant spec said a deployment without source_config.custom_url fails, which task 2.3 changes. Add a MODIFIED requirement that sends assistant requests to the control plane's top-level url, falling back to custom_url, and align task 2.3, the design risk and the proposal. #807 Export, import, compare, and verify prompts and assistants across LangSmith environments Co-Authored-By: Claude Opus 5.5 --- .../design.md | 2 +- .../proposal.md | 2 +- .../specs/assistant/spec.md | 28 +++++++++++++++++++ .../tasks.md | 2 +- 4 files changed, 31 insertions(+), 3 deletions(-) diff --git a/openspec/changes/export-import-prompts-and-assistants/design.md b/openspec/changes/export-import-prompts-and-assistants/design.md index 06299a572..5fa745d6c 100644 --- a/openspec/changes/export-import-prompts-and-assistants/design.md +++ b/openspec/changes/export-import-prompts-and-assistants/design.md @@ -97,7 +97,7 @@ A mise task runs a short Python script that pulls the imported prompt with `incl - [LangSmith may store a pushed model differently from the source] → Import reads the prompt back and reports any difference, and [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8) checks the round trip before the commands are built. - [Pushing a manifest identical to the latest commit may return 409] → Import compares first and pushes nothing when the prompt matches. -- [langstar finds a deployment's Agent Server URL only in `source_config.custom_url` (`resolve_deployment_url` in `cli/src/deployment_utils.rs`), and a deployment built from GitHub has none, so every `langstar assistant` command, the new ones included, fails on such a deployment] → [export-import-prompts-and-assistants Task 2.3 Return the URL of a deployment built from GitHub](tasks.md#L14) changes `resolve_deployment_url` before anyone builds the assistant export, compare, verify and import commands. +- [langstar finds a deployment's Agent Server URL only in `source_config.custom_url` (`resolve_deployment_url` in `cli/src/deployment_utils.rs`), and a deployment built from GitHub has none, so every `langstar assistant` command, the new ones included, fails on such a deployment] → [export-import-prompts-and-assistants Task 2.3 Return the URL of a deployment built from GitHub](tasks.md#L14) makes `resolve_deployment_url` read the deployment's top-level `url`, which the control plane's `Deployment` schema carries, before anyone builds the assistant export, compare, verify and import commands. - [A tag in the destination may already point at another commit] → `--update` moves the tag, and `--dry-run` lists the move. ## Resolved Questions diff --git a/openspec/changes/export-import-prompts-and-assistants/proposal.md b/openspec/changes/export-import-prompts-and-assistants/proposal.md index c58752f7a..0fe88bda4 100644 --- a/openspec/changes/export-import-prompts-and-assistants/proposal.md +++ b/openspec/changes/export-import-prompts-and-assistants/proposal.md @@ -25,7 +25,7 @@ A prompt belongs to a workspace and an assistant belongs to a deployment, but bo ### Modified Capabilities -- `assistant`: adds export, import, compare and verify requirements to `langstar assistant`. +- `assistant`: adds export, import, compare and verify requirements to `langstar assistant`, and lets every assistant command reach a deployment built from GitHub through the deployment's `url`. ## Impact diff --git a/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md b/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md index 124789538..d1eceb102 100644 --- a/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md +++ b/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md @@ -1,3 +1,31 @@ +## MODIFIED Requirements + +### Requirement: Every assistant command targets one deployment + +Every `langstar assistant` subcommand SHALL take a required `--deployment` flag holding a deployment name or ID. Langstar SHALL look the deployment up through the control plane API and send the assistant request to the deployment's `url`, or to its `source_config.custom_url` when `url` is empty. + +#### Scenario: Deployment found by name + +- **WHEN** a user runs `langstar assistant list --deployment my-agent` +- **AND** the workspace has a deployment named `my-agent` +- **THEN** langstar lists the assistants on that deployment + +#### Scenario: Deployment built from GitHub + +- **WHEN** a user runs `langstar assistant list --deployment my-agent` +- **AND** `my-agent` was built from a GitHub repository and has a `url` but no `source_config.custom_url` +- **THEN** langstar lists the assistants on that deployment + +#### Scenario: Deployment not found + +- **WHEN** a user passes a `--deployment` that matches no deployment name or ID in the workspace +- **THEN** langstar exits with an error that names the deployment and suggests `langstar deployment list` + +#### Scenario: Deployment has no URL + +- **WHEN** the deployment exists but has neither a `url` nor a custom URL +- **THEN** langstar exits with an error saying the deployment has no URL + ## ADDED Requirements ### Requirement: Export an assistant to a file diff --git a/openspec/changes/export-import-prompts-and-assistants/tasks.md b/openspec/changes/export-import-prompts-and-assistants/tasks.md index fe7d051d2..4a0be6ef7 100644 --- a/openspec/changes/export-import-prompts-and-assistants/tasks.md +++ b/openspec/changes/export-import-prompts-and-assistants/tasks.md @@ -11,7 +11,7 @@ Each numbered group is one pull request, in the order listed. - [ ] 2.1 Add the destination workspace and the model provider key to `fnox.toml`, the CI workflow and `docs/dev/environment-variables.md`, with placeholders only; verify with `mise run check` and a CI run that reads both. This task waits on codekiln to create a second workspace in the test organization and an OpenAI API key, and to store both in 1Password and the repository's CI secrets. The import tests need the workspace as a place to import into. The key lets the check in 7.3 run each imported structured prompt once against a small OpenAI model, which bills that account on every CI run; it is the one test that shows a copy still returns structured data, where every other test compares text. - [ ] 2.2 Add a destination deployment fixture to `sdk/src/test_utils.rs` and `cli/tests/common/fixtures.rs` that returns a deployment with a reachable URL; verify with an integration test that lists assistants on it -- [ ] 2.3 Make `resolve_deployment_url` in `cli/src/deployment_utils.rs` return the URL of a deployment built from GitHub, which has no `source_config.custom_url`; verify by removing `#[ignore]` from `test_assistant_create_basic` in `cli/tests/assistant_command_test.rs` and running it against the test deployment +- [ ] 2.3 Make `resolve_deployment_url` in `cli/src/deployment_utils.rs` return the deployment's top-level `url` from the control plane, falling back to `source_config.custom_url`, so that a deployment built from GitHub resolves; add a unit test for each case in `sdk/src/deployments.rs`, and verify by removing `#[ignore]` from `test_assistant_create_basic` in `cli/tests/assistant_command_test.rs` and running it against the test deployment ## 3. SDK reads and writes the fields export and import need From 077a113f6ce1254c837dc4914f7ceaa45f8e9724 Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Tue, 13 Oct 2026 18:46:49 -0400 Subject: [PATCH 06/90] =?UTF-8?q?=F0=9F=93=9A=20docs(openspec):=20run=20ve?= =?UTF-8?q?rify's=20checks=20before=20import=20writes=20and=20fix=20the=20?= =?UTF-8?q?SDK=20fields=20tags=20and=20deployments=20need?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add the deployment url to the SDK Deployment and the commit id that the tag endpoints need, add --workspace-id to assistant verify, test the assistant dry run, scope the round-trip goal to the fields the file carries, clear an assistant's description and context on update or report them, and run verify's checks before prompt and assistant import write anything, with Open Question 8 on that choice. #807 Export, import, compare, and verify prompts and assistants across LangSmith environments Co-Authored-By: Claude Opus 5.5 --- .../design.md | 23 ++++++++++++++----- .../proposal.md | 3 ++- .../specs/assistant/spec.md | 23 ++++++++++++++++++- .../specs/prompt/spec.md | 11 +++++++++ .../tasks.md | 16 ++++++------- 5 files changed, 60 insertions(+), 16 deletions(-) diff --git a/openspec/changes/export-import-prompts-and-assistants/design.md b/openspec/changes/export-import-prompts-and-assistants/design.md index 5fa745d6c..4a5435923 100644 --- a/openspec/changes/export-import-prompts-and-assistants/design.md +++ b/openspec/changes/export-import-prompts-and-assistants/design.md @@ -12,7 +12,7 @@ Langstar reads one API key and one workspace at a time, from flags, environment **Goals:** -- Carry every field of a prompt commit and an assistant through a file and back without loss. +- Carry a prompt's manifest and model, its commit description and tags, and its repo details, and an assistant's name, description, graph ID, configuration, context and metadata, through a file and back without loss. - Make a repeated import of the same file safe: it finds its earlier copy and changes nothing until the user asks. - Give users one way to check a copy: the same report shape and exit statuses for both resource types. @@ -49,7 +49,7 @@ One prompt export looks like this: An assistant export has `kind: "assistant"`, a `source` with the deployment, assistant ID and version, an `assistant` object with `name`, `description`, `graph_id`, `config`, `context` and `metadata`, and `requires.prompts`. -The file holds no export time and langstar sorts its keys, so exporting an unchanged resource leaves Git with nothing to commit. Source IDs live only in `source`. +The file holds no export time and langstar sorts its keys, so exporting an unchanged resource leaves Git with nothing to commit. ### Export asks LangSmith for the model @@ -63,13 +63,13 @@ Import writes the destination's IDs to `.destinations.json`, keyed by A prompt that exists in the destination gets a new commit only with `--update`, and its parent is the destination's latest commit, so LangSmith keeps the earlier history. Import sends the file's commit description with each commit it creates. With `--update`, import also sets the destination repo's description, readme, tags and visibility to the file's values through `PATCH /api/v1/repos/{owner}/{repo}` when they differ. When only those repo fields differ and the manifest matches, import updates the repo and adds no commit. -An assistant that exists gets updated only with `--update`, which makes the Agent Server save a new version. The Agent Server merges the metadata an update sends into the assistant's metadata (`AssistantPatch` in `reference/openapi/langchain/agent-server/openapi.json`), so sending the file's metadata alone would keep every key the destination has and the file lacks. Import sends each such key with a null value and then reads the assistant back. [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8) checks against the live server whether a null value removes the key. If it does not, the read-back finds the leftover keys, and import reports them as differences and exits with status 2. Import also sends the file's graph ID, so an assistant whose graph changed at the source moves to that graph. +An assistant that exists gets updated only with `--update`, which makes the Agent Server save a new version. The Agent Server merges the metadata an update sends into the assistant's metadata (`AssistantPatch` in `reference/openapi/langchain/agent-server/openapi.json`), so sending the file's metadata alone would keep every key the destination has and the file lacks. Import sends each such key with a null value and then reads the assistant back. [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8) checks against the live server whether a null value removes the key. If it does not, the read-back finds the leftover keys, and import reports them as differences and exits with status 2. The same `AssistantPatch` takes the description as a string and keeps the destination's description when the request leaves it out, so the OpenAPI spec gives import no documented way to clear one. Import sends the file's configuration and context in full, an empty object when the file holds none, and the same task records whether the Agent Server replaces those objects or merges into them and whether a null or empty description clears one; a description, context or configuration key the file lacks that remains after the update shows up in the read-back as a difference, with status 2. Import also sends the file's graph ID, so an assistant whose graph changed at the source moves to that graph. Without `--update`, import lists the differences and exits with status 2. `--dry-run` reports the action import would take and writes nothing. ### Compare reports differences by JSON path -Compare reads the destination, removes the fields the server sets (IDs, version numbers, timestamps, owner and like counts), and walks both JSON documents. Each difference names its JSON Pointer path and both values. For a prompt, the comparison covers the commit description and the repo's description, readme, tags and visibility as well as the manifest. Import runs the same comparison after it writes, so a field LangSmith or the Agent Server drops shows up as a difference. +Compare reads the destination, removes the fields the server sets (IDs, version numbers, timestamps, owner and like counts), and walks both JSON documents. Compare looks only at the `repo`, `commit` and `assistant` objects. The source environment's workspace, deployment and IDs sit in `source`, so they never show up as differences against a copy. Each difference names its JSON Pointer path and both values. For a prompt, the comparison covers the commit description and the repo's description, readme, tags and visibility as well as the manifest. Import runs the same comparison after it writes, so a field LangSmith or the Agent Server drops shows up as a difference. ### Verify checks what the copy needs to run @@ -77,9 +77,13 @@ Prompt verify looks up each name in `requires.secrets` among the workspace's sec Assistant verify checks the graph with `POST /assistants/search` filtered to the graph ID; every graph on a deployment has a default assistant, so an empty result means the deployment does not serve the graph. It checks each prompt in `requires.prompts`, such as `support-system:prod`, by asking LangSmith for the commit that tag points at (`GET /api/v1/commits/{owner}/{repo}/{tag}`, the request `langstar prompt pull --commit prod` already sends). A 404 means the workspace lacks the prompt or its tag. +### Import runs verify's checks before it writes + +Before it writes anything, import runs the checks verify runs against the destination, because [#807 Export, import, compare, and verify prompts and assistants across LangSmith environments](https://github.com/codekiln/langstar/issues/807) asks import to check the destination's dependencies. Prompt import stops when the destination workspace lacks a secret in `requires.secrets`, and assistant import stops when the deployment does not serve the file's graph or the workspace, from `--workspace-id` or the configured workspace, lacks a prompt in `requires.prompts`. Each names what is missing, writes nothing and exits with status 2, and `--dry-run` reports the same failed checks. A user therefore never gets a copy that cannot run, but has to create the destination's secrets and import the prompts before importing an assistant that refers to them; Open Question 8 asks whether import should write anyway and report the failed checks afterwards. + ### The user names an assistant's prompt references -An assistant refers to a prompt through whatever configuration key its graph reads, so langstar cannot find the references by itself. `assistant export --prompt-ref ` names each one. The assistant's configuration should name each prompt by handle and tag, such as `support-system:prod`, because the same prompt gets a different commit hash in each workspace. Import copies the commit tags to the destination workspace, so `support-system:prod` finds the prompt there too. +An assistant refers to a prompt through whatever configuration key its graph reads, so langstar cannot find the references by itself. `assistant export --prompt-ref ` names each one. The assistant's configuration should name each prompt by handle and tag, such as `support-system:prod`, because the same prompt gets a different commit hash in each workspace. Import copies the commit tags to the destination workspace, so when the graph on the destination deployment asks LangSmith for `support-system:prod`, it gets the same prompt. ### compare, verify and import exit with 2 when they find a difference @@ -97,7 +101,7 @@ A mise task runs a short Python script that pulls the imported prompt with `incl - [LangSmith may store a pushed model differently from the source] → Import reads the prompt back and reports any difference, and [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8) checks the round trip before the commands are built. - [Pushing a manifest identical to the latest commit may return 409] → Import compares first and pushes nothing when the prompt matches. -- [langstar finds a deployment's Agent Server URL only in `source_config.custom_url` (`resolve_deployment_url` in `cli/src/deployment_utils.rs`), and a deployment built from GitHub has none, so every `langstar assistant` command, the new ones included, fails on such a deployment] → [export-import-prompts-and-assistants Task 2.3 Return the URL of a deployment built from GitHub](tasks.md#L14) makes `resolve_deployment_url` read the deployment's top-level `url`, which the control plane's `Deployment` schema carries, before anyone builds the assistant export, compare, verify and import commands. +- [langstar finds a deployment's Agent Server URL only in `source_config.custom_url` (`resolve_deployment_url` in `cli/src/deployment_utils.rs`), and a deployment built from GitHub has none, so every `langstar assistant` command, the new ones included, fails on such a deployment] → [export-import-prompts-and-assistants Task 2.3 Return the URL of a deployment built from GitHub](tasks.md#L14) adds the control plane's top-level `url` to the SDK's `Deployment`, which drops it today, and makes `resolve_deployment_url` use it before anyone builds the assistant export, compare, verify and import commands. - [A tag in the destination may already point at another commit] → `--update` moves the tag, and `--dry-run` lists the move. ## Resolved Questions @@ -154,3 +158,10 @@ None. > Issue #807 asks import to report missing model configurations, so the draft has prompt verify look for a saved model configuration (`GET /api/v1/playground-settings`) whose settings `id` ends in the prompt's `model_provider` class name, such as `ChatOpenAI`. Nothing fails at run time without one, because the prompt's manifest carries the model and its settings; the secret check is the one that finds a prompt that cannot authenticate. Reporting it as a failure would make verify exit with 2 for a prompt that runs. Recommendation: report a missing saved model configuration as a note that leaves the exit status alone. + +### 8 - Should a failed dependency check stop import before it writes? + +> Context from the drafting agent, for question 8. +> Issue #807 asks import to check the destination's dependencies: the secrets a prompt's model reads, and an assistant's graph and referenced prompts. The draft runs those checks before import writes and writes nothing when one fails, so a user never has to find and delete a copy that cannot run. The cost is order: the destination workspace needs its secrets before a prompt import, and its prompts before an assistant import. The other choice writes the copy anyway and reports the failed checks with status 2 afterwards, which lets a user import in any order and fix the destination later. Recommendation: stop before writing. + + diff --git a/openspec/changes/export-import-prompts-and-assistants/proposal.md b/openspec/changes/export-import-prompts-and-assistants/proposal.md index 0fe88bda4..5a1731466 100644 --- a/openspec/changes/export-import-prompts-and-assistants/proposal.md +++ b/openspec/changes/export-import-prompts-and-assistants/proposal.md @@ -15,6 +15,7 @@ A prompt belongs to a workspace and an assistant belongs to a deployment, but bo - `langstar prompt compare ` reports the differences between the file and a prompt in a workspace. `langstar prompt verify ` also checks that the workspace has the secrets the prompt's model reads, and notes when the workspace has no saved model configuration for the model's provider. - `langstar assistant export`, `import`, `compare` and `verify` do the same for assistants on a deployment. The file holds the name, description, graph ID, configuration, context, metadata and any prompt names the assistant refers to. `verify` checks that the destination deployment has the graph and that the destination workspace has the referenced prompts. - Import records each destination ID (the new commit hash, or the new assistant ID) in a separate file next to the export, so importing the same file again finds the copy it made earlier instead of creating another one. +- Import runs the checks of `verify` before it writes, and writes nothing when one fails. - `compare`, `verify` and `import` share one exit status scheme and print a JSON report with `-f json`. ## Capabilities @@ -29,7 +30,7 @@ A prompt belongs to a workspace and an assistant belongs to a deployment, but bo ## Impact -- SDK: `sdk/src/prompts.rs` gains a commit read that asks for the model, a commit description on commit create, a repo update and calls for prompt tags. `sdk/src/assistants.rs` gains the `context` and `version` fields, `assistant_id` and `if_exists` on create, `graph_id` on update, and a search by graph ID. +- SDK: `sdk/src/prompts.rs` gains a commit read that asks for the model, a commit description on commit create, the new commit's `id`, a repo update, and calls that list, create and move commit tags. `sdk/src/assistants.rs` gains the `context` and `version` fields, `assistant_id` and `if_exists` on create, `graph_id` on update, and a search by graph ID. `sdk/src/deployments.rs` gains the deployment's `url`. - CLI: new subcommands in `cli/src/commands/prompt.rs` and `cli/src/commands/assistant.rs`, and a shared module for the export file, the destination record, comparison and the exit status. `resolve_deployment_url` in `cli/src/deployment_utils.rs` also returns the Agent Server URL of a deployment built from GitHub, which every `langstar assistant` command needs. - Tests need a second workspace, a second deployment and one model provider key for the structured output check. codekiln creates the workspace and the key, and langstar's tests read them through `fnox.toml` and the CI workflow; the `TestDeploymentConfig` fixture creates the deployment on the first run and the tests reuse it after that. - Docs: a new usage page on moving prompts and assistants between environments. diff --git a/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md b/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md index d1eceb102..33dfaa13e 100644 --- a/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md +++ b/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md @@ -69,6 +69,17 @@ Every `langstar assistant` subcommand SHALL take a required `--deployment` flag - **THEN** langstar creates the assistant and prints its new ID - **AND** `support-bot.destinations.json` holds the ID of the `my-agent-prod` deployment, with the new assistant ID under it +### Requirement: Check the destination before importing an assistant + +Before it writes, `langstar assistant import` SHALL run the graph check and the prompt check of `assistant verify`, reading prompts from `--workspace-id` or the configured workspace. When a check fails it SHALL write nothing, name each missing graph and prompt, and exit with status 2. + +#### Scenario: Referenced prompt missing at import + +- **WHEN** the file lists `support-system:prod` under `requires.prompts` +- **AND** the workspace has no `support-system` prompt with a `prod` tag +- **AND** a user runs `langstar assistant import support-bot.json --deployment my-agent-prod` +- **THEN** langstar creates no assistant, reports `support-system:prod` as missing and exits with status 2 + ### Requirement: Find the earlier copy instead of creating another When the destination record holds an assistant ID for the deployment, `langstar assistant import` SHALL compare the file with that assistant and create nothing. It SHALL exit with status 0 when they match, and list the differences and exit with status 2 when they differ, unless the user passes `--update`. When the recorded assistant no longer exists, it SHALL exit with status 1 and say so. @@ -85,7 +96,7 @@ When the destination record holds an assistant ID for the deployment, `langstar ### Requirement: Update an existing assistant only on request -`langstar assistant import --update` SHALL update the recorded assistant, or the assistant named by `--assistant-id `, with the file's name, description, graph ID, configuration, context and metadata. `--assistant-id` SHALL take precedence over the destination record. It SHALL remove each metadata key the destination assistant has and the file lacks. When such a key remains after the update, it SHALL list that key as a difference and exit with status 2. +`langstar assistant import --update` SHALL update the recorded assistant, or the assistant named by `--assistant-id `, with the file's name, description, graph ID, configuration, context and metadata. `--assistant-id` SHALL take precedence over the destination record. #### Scenario: Update the recorded assistant @@ -98,6 +109,10 @@ When the destination record holds an assistant ID for the deployment, `langstar - **WHEN** the user runs `langstar assistant import support-bot.json --deployment my-agent-prod --update --assistant-id ` - **THEN** langstar updates that assistant and records its ID for `my-agent-prod` +### Requirement: Remove on update what the file lacks + +`langstar assistant import --update` SHALL remove each metadata key the destination assistant has and the file lacks, and clear a description or context the file lacks. When such a key or field remains after the update, it SHALL list it as a difference and exit with status 2. + #### Scenario: Remove a metadata key the file lacks - **WHEN** the recorded assistant's metadata is `{"kept": 1, "stale": 2}` @@ -111,6 +126,12 @@ When the destination record holds an assistant ID for the deployment, `langstar - **WHEN** the Agent Server keeps `stale` in the assistant's metadata after the update - **THEN** langstar lists `/metadata/stale` as a difference and exits with status 2 +#### Scenario: Deployment keeps a description the file lacks + +- **WHEN** the recorded assistant has a description and the file has none +- **AND** the Agent Server keeps that description after the update +- **THEN** langstar lists `/description` as a difference and exits with status 2 + ### Requirement: Dry run an assistant import `langstar assistant import --dry-run` SHALL report whether it would create the assistant, update it, or leave it unchanged, list the differences, and change nothing on the deployment or in the destination record. diff --git a/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md index dc94a35a7..5b07c2286 100644 --- a/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md +++ b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md @@ -62,6 +62,17 @@ The export file SHALL list, under `requires.secrets`, the name of every secret t - **WHEN** a user runs `langstar prompt import my-prompt.json` without `--workspace-id` - **THEN** langstar exits with status 1 and an error asking for `--workspace-id` +### Requirement: Check the destination before importing a prompt + +Before it writes, `langstar prompt import` SHALL run the checks of `prompt verify` against the destination workspace. When the workspace lacks a secret in `requires.secrets` it SHALL write nothing, name each missing secret and exit with status 2. A missing saved model configuration SHALL be a note, as in `prompt verify`. + +#### Scenario: Secret missing at import + +- **WHEN** the file lists `OPENAI_API_KEY` under `requires.secrets` +- **AND** the destination workspace has no secret named `OPENAI_API_KEY` +- **AND** a user runs `langstar prompt import my-prompt.json --workspace-id ` +- **THEN** langstar creates no prompt, reports `OPENAI_API_KEY` as missing and exits with status 2 + ### Requirement: Import a prompt that does not exist yet When the destination workspace has no prompt with the file's handle, `langstar prompt import` SHALL create the prompt with the file's repo details, push the file's manifest and commit description as its first commit, and create the file's commit tags on that commit. diff --git a/openspec/changes/export-import-prompts-and-assistants/tasks.md b/openspec/changes/export-import-prompts-and-assistants/tasks.md index 4a0be6ef7..055ccba5c 100644 --- a/openspec/changes/export-import-prompts-and-assistants/tasks.md +++ b/openspec/changes/export-import-prompts-and-assistants/tasks.md @@ -5,18 +5,18 @@ Each numbered group is one pull request, in the order listed. ## 1. Check the API round trip before building on it - [ ] 1.1 Extract the commit get, commit create, repo update (`PATCH /api/v1/repos/{owner}/{repo}`), repo tags (`/api/v1/repos/{owner}/{repo}/tags` and `/api/v1/repos/{owner}/{repo}/tags/{tag_name}`), workspace secrets and playground settings endpoints from `reference/openapi/langchain/langsmith/openapi.json` into `reference/api-specs/langsmith/` and add their jq queries to `FRAGMENTS.md`; verify by rerunning each query and getting the same file -- [ ] 1.2 Write `reference/research/807-openapi-validation.md`, recording from the live API: the manifest returned with `include=model` for a structured prompt with a model and the outer `id` of that manifest, whether pushing that manifest to a second workspace returns the same model, what pushing an identical manifest returns, what the Agent Server returns for `context`, whether an Agent Server assistant update that sends a metadata key with a null value removes that key, and what `POST /api/v1/prompts/invoke_prompt` returns for a prompt with a model; verify that every claim in the report names the request that produced it +- [ ] 1.2 Write `reference/research/807-openapi-validation.md`, recording from the live API: the manifest returned with `include=model` for a structured prompt with a model and the outer `id` of that manifest, whether pushing that manifest to a second workspace returns the same model, what pushing an identical manifest returns, what the Agent Server returns for `context`, whether an Agent Server assistant update that sends a metadata key with a null value removes that key, whether an update that sends `config` or `context` replaces the assistant's object or merges into it, whether an update can clear an assistant's description, and what `POST /api/v1/prompts/invoke_prompt` returns for a prompt with a model; verify that every claim in the report names the request that produced it ## 2. Test environments -- [ ] 2.1 Add the destination workspace and the model provider key to `fnox.toml`, the CI workflow and `docs/dev/environment-variables.md`, with placeholders only; verify with `mise run check` and a CI run that reads both. This task waits on codekiln to create a second workspace in the test organization and an OpenAI API key, and to store both in 1Password and the repository's CI secrets. The import tests need the workspace as a place to import into. The key lets the check in 7.3 run each imported structured prompt once against a small OpenAI model, which bills that account on every CI run; it is the one test that shows a copy still returns structured data, where every other test compares text. +- [ ] 2.1 Add the destination workspace and the model provider key to `fnox.toml`, the CI workflow and `docs/dev/environment-variables.md`, with placeholders only; verify with `mise run check` and a CI run that reads both. This task waits on codekiln to create a second workspace in the test organization and an OpenAI API key, and to store both in 1Password and the repository's CI secrets and the key as the destination workspace's `OPENAI_API_KEY` secret. The import tests need the workspace as a place to import into, and its secret lets prompt import's secret check pass for a prompt with an OpenAI model. The key lets the check in 7.3 run each imported structured prompt once against a small OpenAI model, which bills that account on every CI run; it is the one test that shows a copy still returns structured data, where every other test compares text. - [ ] 2.2 Add a destination deployment fixture to `sdk/src/test_utils.rs` and `cli/tests/common/fixtures.rs` that returns a deployment with a reachable URL; verify with an integration test that lists assistants on it -- [ ] 2.3 Make `resolve_deployment_url` in `cli/src/deployment_utils.rs` return the deployment's top-level `url` from the control plane, falling back to `source_config.custom_url`, so that a deployment built from GitHub resolves; add a unit test for each case in `sdk/src/deployments.rs`, and verify by removing `#[ignore]` from `test_assistant_create_basic` in `cli/tests/assistant_command_test.rs` and running it against the test deployment +- [ ] 2.3 Add `url` to `Deployment` in `sdk/src/deployments.rs`, with a method that returns `url` and falls back to `source_config.custom_url`, and make `resolve_deployment_url` in `cli/src/deployment_utils.rs` call that method, so that a deployment built from GitHub resolves; add a unit test for each case in `sdk/src/deployments.rs`, and verify by removing `#[ignore]` from `test_assistant_create_basic` in `cli/tests/assistant_command_test.rs` and running it against the test deployment ## 3. SDK reads and writes the fields export and import need - [ ] 3.1 Add a commit read that sends `include=model` and returns `description`, `model_config` and `model_provider` to `sdk/src/prompts.rs`; verify with an httpmock test that asserts the query string and the parsed fields, and an integration test that reads a prompt with a model -- [ ] 3.2 Add listing, creating and moving prompt tags to `sdk/src/prompts.rs`, moving a tag with `PATCH /api/v1/repos/{owner}/{repo}/tags/{tag_name}`; verify with httpmock tests and an integration test that creates a tag on one commit, moves it to a second commit, and reads it back through the SDK +- [ ] 3.2 Add the commit's `id` to `CommitData` in `sdk/src/prompts.rs`, and add calls that list, create and move commit tags, which name their commit by that `id` in `POST /api/v1/repos/{owner}/{repo}/tags` and `PATCH /api/v1/repos/{owner}/{repo}/tags/{tag_name}`; verify with httpmock tests that assert `commit_id` in each request body, and an integration test that pushes two commits, tags the first, moves the tag to the second, and reads the tag back through the SDK - [ ] 3.3 Add `context` and `version` to `Assistant`, `context` and `graph_id` to `UpdateAssistantRequest`, `context` to `CreateAssistantRequest`, `assistant_id` and `if_exists` to `CreateAssistantRequest`, and a `graph_id` filter to `AssistantSearchRequest`, in `sdk/src/assistants.rs`; verify with httpmock tests of the request bodies and an integration test that creates an assistant with context, reads it back with its version, finds it by graph ID, and updates its graph ID and reads the new graph ID back - [ ] 3.4 Add `description` to `CommitRequest`, and a repo update that sends `PATCH /api/v1/repos/{owner}/{repo}` with `description`, `readme`, `tags` and `is_public`, to `sdk/src/prompts.rs`; verify with httpmock tests of the request bodies and an integration test that pushes a commit with a description, updates the repo's description, readme, tags and visibility, and reads each back @@ -40,7 +40,7 @@ Each numbered group is one pull request, in the order listed. ## 7. prompt import -- [ ] 7.1 Add `langstar prompt import` with required `--workspace-id`, creating the repo, the commit and its tags, then reading the commit back; verify with integration tests that export from the test workspace, import into the destination workspace, and compare the full source and destination commits through the SDK with the model included, the commit description among them, and the repo's description, readme, tags and visibility, for every prompt shape from 5.1 +- [ ] 7.1 Add `langstar prompt import` with required `--workspace-id`, running the checks of `prompt verify` from 6.2 before it writes, creating the repo, the commit and its tags, then reading the commit back; verify with integration tests that export from the test workspace, import into the destination workspace, and compare the full source and destination commits through the SDK with the model included, the commit description among them, and the repo's description, readme, tags and visibility, for every prompt shape from 5.1, and that a file whose `requires.secrets` names a secret the destination workspace lacks creates no prompt, names the secret and exits with status 2 - [ ] 7.2 Add `--update`, `--dry-run`, `--record` and the destination record; verify with integration tests that a repeated import creates no commit, that a changed destination is refused without `--update`, that an import after someone pushed a commit to the destination since the recorded import says the destination changed since the last import and exits with status 2, that `--update` adds one commit with the right parent and the file's commit description, that `--update` against a destination whose repo description, readme, tags or visibility differ while the manifest matches updates the repo and adds no commit, and that `--dry-run` writes nothing - [ ] 7.3 Add the structured output check as a mise task that pulls the imported prompt with `include_model=True`, invokes it and validates the result against the schema; verify by running it from the import integration test for both methods - [ ] 7.4 Document prompt import, the workspace scope of prompts and setting up destination credentials; verify that each documented command runs as written @@ -53,13 +53,13 @@ Each numbered group is one pull request, in the order listed. ## 9. assistant compare and verify - [ ] 9.1 Add `langstar assistant compare` with `--assistant-id`, and the lookup of the recorded assistant in the destination record from 4.1; verify with an integration test that exports an assistant, changes its metadata through the SDK, runs compare with `--assistant-id` set to that assistant, and asserts the reported path and exit status 2, and a unit test that compare reads the assistant ID from a destination record written by the 4.1 types -- [ ] 9.2 Add `langstar assistant verify` with `--assistant-id`, the graph check and the prompt check, which reads `GET /api/v1/commits/{owner}/{repo}/{tag}`; verify with integration tests that run verify with `--assistant-id` against an exported assistant, one with a file whose graph ID the deployment lacks and one with a missing prompt tag +- [ ] 9.2 Add `langstar assistant verify` with `--assistant-id`, `--workspace-id` for the workspace that holds the prompts, the graph check, and the prompt check, which reads `GET /api/v1/commits/{owner}/{repo}/{tag}`; verify with integration tests that run verify with `--assistant-id` against an exported assistant, one with a file whose graph ID the deployment lacks, one with a missing prompt tag, and one that passes `--workspace-id` set to the destination workspace - [ ] 9.3 Document assistant compare and verify; verify that each documented command runs as written ## 10. assistant import -- [ ] 10.1 Add `langstar assistant import` with creation, read-back, the destination record and `--record`; verify with an integration test that imports into the destination deployment and checks name, graph ID, configuration, context, metadata and prompt references through the Agent Server API, and that `--record ` writes the destination record to that path -- [ ] 10.2 Add `--update`, `--assistant-id` and `--dry-run`; verify with integration tests that a repeated import creates no assistant, that `--update` changes the recorded one, its graph ID among the fields, that `--update` of an assistant with metadata `{"kept": 1, "stale": 2}` from a file with `{"kept": 1}` leaves `{"kept": 1}` or reports `/metadata/stale` and exits with status 2, whichever task 1.2 found the server does, and that a deleted recorded assistant exits with status 1 +- [ ] 10.1 Add `langstar assistant import` with `--workspace-id`, the graph check and prompt check of `assistant verify` from 9.2 run before it writes, creation, read-back, the destination record and `--record`; verify with an integration test that creates a tagged prompt in the test workspace, imports an assistant that refers to it into the destination deployment and checks name, graph ID, configuration, context, metadata and prompt references through the Agent Server API, that `--record ` writes the destination record to that path, and that a file whose graph ID the deployment lacks and a file with a missing prompt tag each create no assistant and exit with status 2 +- [ ] 10.2 Add `--update`, `--assistant-id` and `--dry-run`; verify with integration tests that a repeated import creates no assistant, that `--update` changes the recorded one, its graph ID among the fields, that `--update` of an assistant with metadata `{"kept": 1, "stale": 2}` from a file with `{"kept": 1}` leaves `{"kept": 1}` or reports `/metadata/stale` and exits with status 2, whichever task 1.2 found the server does, that `--update` of an assistant with a description and context from a file with neither clears both or reports `/description` and `/context` and exits with status 2, whichever task 1.2 found the server does, that a deleted recorded assistant exits with status 1, and that `--dry-run` creates no assistant, changes none and leaves the destination record as it was - [ ] 10.3 Add an end-to-end test that imports prompts into the destination workspace and then an assistant that refers to them onto the destination deployment, and runs `assistant verify`; verify that it passes and that it fails after deleting one prompt tag - [ ] 10.4 Document assistant import, the deployment scope of assistants, and how export and import relate to the `create`, `get` and `update` commands of [#728 Implement CRUD command structure (create, get, update) for prompt commands](https://github.com/codekiln/langstar/issues/728); verify that each documented command runs as written From 21854ea5294e29559f9b133e8e7897c176db3ae5 Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Tue, 13 Oct 2026 18:56:55 -0400 Subject: [PATCH 07/90] =?UTF-8?q?=F0=9F=93=9A=20docs(openspec):=20build=20?= =?UTF-8?q?the=20destination=20deployment=20from=20its=20own=20graph=20con?= =?UTF-8?q?fig=20and=20move=20commit=20tags=20on=20import?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Swap tasks 2.2 and 2.3 so the destination deployment fixture comes after the deployment URL it needs, and build that deployment from its own langgraph.json so the fixture never returns the source deployment. Treat a commit tag that is missing or points elsewhere as a difference that --update moves. Lead the assistant update paragraph with what the Agent Server keeps. Test that prompt import without --workspace-id fails while LANGSMITH_WORKSPACE_ID is set. #807 Export, import, compare, and verify prompts and assistants across LangSmith environments Co-Authored-By: Claude Opus 5.5 --- .../design.md | 10 +++++----- .../proposal.md | 2 +- .../specs/prompt/spec.md | 15 ++++++++++++++- .../export-import-prompts-and-assistants/tasks.md | 8 ++++---- 4 files changed, 24 insertions(+), 11 deletions(-) diff --git a/openspec/changes/export-import-prompts-and-assistants/design.md b/openspec/changes/export-import-prompts-and-assistants/design.md index 4a5435923..a62c4fff3 100644 --- a/openspec/changes/export-import-prompts-and-assistants/design.md +++ b/openspec/changes/export-import-prompts-and-assistants/design.md @@ -61,15 +61,15 @@ Import writes the destination's IDs to `.destinations.json`, keyed by ### Import changes an existing resource only with --update -A prompt that exists in the destination gets a new commit only with `--update`, and its parent is the destination's latest commit, so LangSmith keeps the earlier history. Import sends the file's commit description with each commit it creates. With `--update`, import also sets the destination repo's description, readme, tags and visibility to the file's values through `PATCH /api/v1/repos/{owner}/{repo}` when they differ. When only those repo fields differ and the manifest matches, import updates the repo and adds no commit. +A prompt that exists in the destination gets a new commit only with `--update`, and its parent is the destination's latest commit, so LangSmith keeps the earlier history. Import sends the file's commit description with each commit it creates. With `--update`, import also sets the destination repo's description, readme, tags and visibility to the file's values through `PATCH /api/v1/repos/{owner}/{repo}` when they differ. When only those repo fields differ and the manifest matches, import updates the repo and adds no commit. A commit tag in the file that the destination lacks, or that points at another commit there, also counts as a difference; with `--update`, import creates or moves the tag to the destination commit that matches the file, and adds no commit when the latest commit already matches. -An assistant that exists gets updated only with `--update`, which makes the Agent Server save a new version. The Agent Server merges the metadata an update sends into the assistant's metadata (`AssistantPatch` in `reference/openapi/langchain/agent-server/openapi.json`), so sending the file's metadata alone would keep every key the destination has and the file lacks. Import sends each such key with a null value and then reads the assistant back. [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8) checks against the live server whether a null value removes the key. If it does not, the read-back finds the leftover keys, and import reports them as differences and exits with status 2. The same `AssistantPatch` takes the description as a string and keeps the destination's description when the request leaves it out, so the OpenAPI spec gives import no documented way to clear one. Import sends the file's configuration and context in full, an empty object when the file holds none, and the same task records whether the Agent Server replaces those objects or merges into them and whether a null or empty description clears one; a description, context or configuration key the file lacks that remains after the update shows up in the read-back as a difference, with status 2. Import also sends the file's graph ID, so an assistant whose graph changed at the source moves to that graph. +The Agent Server keeps the name, graph ID and description an update leaves out, and merges the metadata an update sends into the metadata the assistant already has, as `AssistantPatch` in `reference/openapi/langchain/agent-server/openapi.json` documents. So with `--update`, import asks for each removal and then checks that it happened. It sends the file's name, graph ID, configuration and context in full, an empty object for a configuration or context the file lacks, and a null value for each metadata key the destination has and the file lacks. `AssistantPatch` documents no way to clear a description, so [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8) tries a null and an empty description against the live server, along with a null metadata value and a replaced configuration and context. Import then reads the assistant back, and any key or field the file lacks that is still there shows up as a difference, with exit status 2. The Agent Server saves each update as a new version of the assistant. Without `--update`, import lists the differences and exits with status 2. `--dry-run` reports the action import would take and writes nothing. ### Compare reports differences by JSON path -Compare reads the destination, removes the fields the server sets (IDs, version numbers, timestamps, owner and like counts), and walks both JSON documents. Compare looks only at the `repo`, `commit` and `assistant` objects. The source environment's workspace, deployment and IDs sit in `source`, so they never show up as differences against a copy. Each difference names its JSON Pointer path and both values. For a prompt, the comparison covers the commit description and the repo's description, readme, tags and visibility as well as the manifest. Import runs the same comparison after it writes, so a field LangSmith or the Agent Server drops shows up as a difference. +Compare reads the destination, removes the fields the server sets (IDs, version numbers, timestamps, owner and like counts), and walks both JSON documents. Compare looks only at the `repo`, `commit` and `assistant` objects. The source environment's workspace, deployment and IDs sit in `source`, so they never show up as differences against a copy. Each difference names its JSON Pointer path and both values. For a prompt, the comparison covers the commit description, the commit tags and the commit each one points at, and the repo's description, readme, tags and visibility as well as the manifest. Import runs the same comparison after it writes, so a field LangSmith or the Agent Server drops shows up as a difference. ### Verify checks what the copy needs to run @@ -101,7 +101,7 @@ A mise task runs a short Python script that pulls the imported prompt with `incl - [LangSmith may store a pushed model differently from the source] → Import reads the prompt back and reports any difference, and [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8) checks the round trip before the commands are built. - [Pushing a manifest identical to the latest commit may return 409] → Import compares first and pushes nothing when the prompt matches. -- [langstar finds a deployment's Agent Server URL only in `source_config.custom_url` (`resolve_deployment_url` in `cli/src/deployment_utils.rs`), and a deployment built from GitHub has none, so every `langstar assistant` command, the new ones included, fails on such a deployment] → [export-import-prompts-and-assistants Task 2.3 Return the URL of a deployment built from GitHub](tasks.md#L14) adds the control plane's top-level `url` to the SDK's `Deployment`, which drops it today, and makes `resolve_deployment_url` use it before anyone builds the assistant export, compare, verify and import commands. +- [langstar finds a deployment's Agent Server URL only in `source_config.custom_url` (`resolve_deployment_url` in `cli/src/deployment_utils.rs`), and a deployment built from GitHub has none, so every `langstar assistant` command, the new ones included, fails on such a deployment] → [export-import-prompts-and-assistants Task 2.2 Return the URL of a deployment built from GitHub](tasks.md#L13) adds the control plane's top-level `url` to the SDK's `Deployment`, which drops it today, and makes `resolve_deployment_url` use it before anyone builds the assistant export, compare, verify and import commands. - [A tag in the destination may already point at another commit] → `--update` moves the tag, and `--dry-run` lists the move. ## Resolved Questions @@ -141,7 +141,7 @@ None. ### 5 - Which second workspace and second deployment should the tests use? > Context from the drafting agent, for question 5. -> The tests need a destination workspace and a destination deployment. The draft adds `LANGSTAR_TEST_DEST_WORKSPACE_ID`, a second workspace in the test organization so the existing API key works, and a second test deployment built by the existing `TestDeploymentConfig` fixture under its own name prefix. A second deployment costs build time and hosting in every CI run that creates it. Recommendation: a second workspace in the same organization, which codekiln creates, and one destination deployment that the `TestDeploymentConfig` fixture creates on the first run and the tests reuse after that. +> The tests need a destination workspace and a destination deployment. The draft adds `LANGSTAR_TEST_DEST_WORKSPACE_ID`, a second workspace in the test organization so the existing API key works, and a second test deployment that the existing `TestDeploymentConfig` fixture builds from its own `langgraph.json`, because the control plane allows one deployment per repository and graph config file. A second deployment costs build time and hosting in every CI run that creates it. Recommendation: a second workspace in the same organization, which codekiln creates, and one destination deployment that the `TestDeploymentConfig` fixture creates on the first run and the tests reuse after that. diff --git a/openspec/changes/export-import-prompts-and-assistants/proposal.md b/openspec/changes/export-import-prompts-and-assistants/proposal.md index 5a1731466..4738febc7 100644 --- a/openspec/changes/export-import-prompts-and-assistants/proposal.md +++ b/openspec/changes/export-import-prompts-and-assistants/proposal.md @@ -11,7 +11,7 @@ A prompt belongs to a workspace and an assistant belongs to a deployment, but bo ## What Changes - `langstar prompt export ` writes one JSON file holding a saved prompt commit as LangSmith returns it with the model included: message templates, input variables, output schema, structured output method, model binding and commit description, plus the repo's description, readme, tags and visibility, the commit tags, and where it came from. -- `langstar prompt import --workspace-id ` creates the prompt in the destination workspace, or adds a commit to an existing prompt and updates its repo details when the user passes `--update`. `--dry-run` shows what it would do. After writing, it reads the prompt back and reports any difference. +- `langstar prompt import --workspace-id ` creates the prompt in the destination workspace, or adds a commit to an existing prompt, updates its repo details and moves its commit tags when the user passes `--update`. `--dry-run` shows what it would do. After writing, it reads the prompt back and reports any difference. - `langstar prompt compare ` reports the differences between the file and a prompt in a workspace. `langstar prompt verify ` also checks that the workspace has the secrets the prompt's model reads, and notes when the workspace has no saved model configuration for the model's provider. - `langstar assistant export`, `import`, `compare` and `verify` do the same for assistants on a deployment. The file holds the name, description, graph ID, configuration, context, metadata and any prompt names the assistant refers to. `verify` checks that the destination deployment has the graph and that the destination workspace has the referenced prompts. - Import records each destination ID (the new commit hash, or the new assistant ID) in a separate file next to the export, so importing the same file again finds the copy it made earlier instead of creating another one. diff --git a/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md index 5b07c2286..5d1e6dd66 100644 --- a/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md +++ b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md @@ -60,6 +60,7 @@ The export file SHALL list, under `requires.secrets`, the name of every secret t #### Scenario: Missing destination workspace - **WHEN** a user runs `langstar prompt import my-prompt.json` without `--workspace-id` +- **AND** `LANGSMITH_WORKSPACE_ID` is set - **THEN** langstar exits with status 1 and an error asking for `--workspace-id` ### Requirement: Check the destination before importing a prompt @@ -102,7 +103,7 @@ When the destination prompt exists and its latest commit differs from the file, #### Scenario: Destination already matches -- **WHEN** the destination prompt's latest commit matches the file +- **WHEN** the destination prompt's latest commit and its commit tags match the file - **THEN** langstar pushes nothing, reports that the prompt is unchanged and exits with status 0 ### Requirement: Update the repo details only with --update @@ -117,6 +118,18 @@ When the destination repo's description, readme, tags or visibility differ from - **THEN** langstar sets the destination repo's readme to the file's readme and adds no commit - **AND** exits with status 0 +### Requirement: Move commit tags only with --update + +When a commit tag in the file is missing in the destination or points at a different commit, `langstar prompt import` SHALL list the tag as a difference and exit with status 2. With `--update` it SHALL create or move the tag to the destination commit that matches the file, and add no commit when the latest commit already matches. + +#### Scenario: Destination tag points at an older commit + +- **WHEN** the destination prompt's latest commit matches the file +- **AND** the destination's `prod` tag points at an earlier commit +- **AND** the user runs `langstar prompt import my-prompt.json --workspace-id --update` +- **THEN** langstar moves `prod` to the latest commit and adds no commit +- **AND** exits with status 0 + ### Requirement: Record the destination commit After a successful import, langstar SHALL record the destination workspace, handle and commit hash in a destination record file, `.destinations.json` next to the export unless `--record ` names another. Each import SHALL add or replace only the entry for its own destination workspace. diff --git a/openspec/changes/export-import-prompts-and-assistants/tasks.md b/openspec/changes/export-import-prompts-and-assistants/tasks.md index 055ccba5c..125937c08 100644 --- a/openspec/changes/export-import-prompts-and-assistants/tasks.md +++ b/openspec/changes/export-import-prompts-and-assistants/tasks.md @@ -10,8 +10,8 @@ Each numbered group is one pull request, in the order listed. ## 2. Test environments - [ ] 2.1 Add the destination workspace and the model provider key to `fnox.toml`, the CI workflow and `docs/dev/environment-variables.md`, with placeholders only; verify with `mise run check` and a CI run that reads both. This task waits on codekiln to create a second workspace in the test organization and an OpenAI API key, and to store both in 1Password and the repository's CI secrets and the key as the destination workspace's `OPENAI_API_KEY` secret. The import tests need the workspace as a place to import into, and its secret lets prompt import's secret check pass for a prompt with an OpenAI model. The key lets the check in 7.3 run each imported structured prompt once against a small OpenAI model, which bills that account on every CI run; it is the one test that shows a copy still returns structured data, where every other test compares text. -- [ ] 2.2 Add a destination deployment fixture to `sdk/src/test_utils.rs` and `cli/tests/common/fixtures.rs` that returns a deployment with a reachable URL; verify with an integration test that lists assistants on it -- [ ] 2.3 Add `url` to `Deployment` in `sdk/src/deployments.rs`, with a method that returns `url` and falls back to `source_config.custom_url`, and make `resolve_deployment_url` in `cli/src/deployment_utils.rs` call that method, so that a deployment built from GitHub resolves; add a unit test for each case in `sdk/src/deployments.rs`, and verify by removing `#[ignore]` from `test_assistant_create_basic` in `cli/tests/assistant_command_test.rs` and running it against the test deployment +- [ ] 2.2 Add `url` to `Deployment` in `sdk/src/deployments.rs`, with a method that returns `url` and falls back to `source_config.custom_url`, and make `resolve_deployment_url` in `cli/src/deployment_utils.rs` call that method, so that a deployment built from GitHub resolves; add a unit test for each case in `sdk/src/deployments.rs`, and verify by removing `#[ignore]` from `test_assistant_create_basic` in `cli/tests/assistant_command_test.rs` and running it against the test deployment +- [ ] 2.3 Add a destination deployment fixture to `sdk/src/test_utils.rs` and `cli/tests/common/fixtures.rs` that builds the test graph from its own `langgraph.json` under `tests/fixtures/`, because the control plane allows one deployment per repository and graph config file and `find_reusable_deployment` would otherwise hand back the source deployment; verify with an integration test that asserts the destination and source deployment IDs differ and lists assistants on the destination deployment through the URL that 2.2 resolves ## 3. SDK reads and writes the fields export and import need @@ -40,8 +40,8 @@ Each numbered group is one pull request, in the order listed. ## 7. prompt import -- [ ] 7.1 Add `langstar prompt import` with required `--workspace-id`, running the checks of `prompt verify` from 6.2 before it writes, creating the repo, the commit and its tags, then reading the commit back; verify with integration tests that export from the test workspace, import into the destination workspace, and compare the full source and destination commits through the SDK with the model included, the commit description among them, and the repo's description, readme, tags and visibility, for every prompt shape from 5.1, and that a file whose `requires.secrets` names a secret the destination workspace lacks creates no prompt, names the secret and exits with status 2 -- [ ] 7.2 Add `--update`, `--dry-run`, `--record` and the destination record; verify with integration tests that a repeated import creates no commit, that a changed destination is refused without `--update`, that an import after someone pushed a commit to the destination since the recorded import says the destination changed since the last import and exits with status 2, that `--update` adds one commit with the right parent and the file's commit description, that `--update` against a destination whose repo description, readme, tags or visibility differ while the manifest matches updates the repo and adds no commit, and that `--dry-run` writes nothing +- [ ] 7.1 Add `langstar prompt import` with required `--workspace-id`, running the checks of `prompt verify` from 6.2 before it writes, creating the repo, the commit and its tags, then reading the commit back; verify with integration tests that export from the test workspace, import into the destination workspace, and compare the full source and destination commits through the SDK with the model included, the commit description among them, and the repo's description, readme, tags and visibility, for every prompt shape from 5.1, that a file whose `requires.secrets` names a secret the destination workspace lacks creates no prompt, names the secret and exits with status 2, and a CLI test that runs import without `--workspace-id` while `LANGSMITH_WORKSPACE_ID` is set and asserts that it exits with status 1, names the missing flag and creates no prompt +- [ ] 7.2 Add `--update`, `--dry-run`, `--record` and the destination record; verify with integration tests that a repeated import creates no commit, that a changed destination is refused without `--update`, that an import after someone pushed a commit to the destination since the recorded import says the destination changed since the last import and exits with status 2, that `--update` adds one commit with the right parent and the file's commit description, that `--update` against a destination whose repo description, readme, tags or visibility differ while the manifest matches updates the repo and adds no commit, that `--update` against a destination whose `prod` tag points at an earlier commit while the manifest matches moves the tag and adds no commit, that `--dry-run` lists the move, and that `--dry-run` writes nothing - [ ] 7.3 Add the structured output check as a mise task that pulls the imported prompt with `include_model=True`, invokes it and validates the result against the schema; verify by running it from the import integration test for both methods - [ ] 7.4 Document prompt import, the workspace scope of prompts and setting up destination credentials; verify that each documented command runs as written From 257617fb0a9d242c04dc8c95c3480f8af202b4da Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Tue, 13 Oct 2026 19:05:56 -0400 Subject: [PATCH 08/90] =?UTF-8?q?=F0=9F=93=9A=20docs(openspec):=20find=20t?= =?UTF-8?q?ag=20commits=20through=20the=20commit=20list=20and=20take=20the?= =?UTF-8?q?=20model=20class=20from=20the=20manifest?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Plan an SDK call for the commit list so the tag move can name a commit it did not create, take requires.model_provider from the chat model constructor's id, add tests for --commit prod and --assistant-id, and reword two design sentences. #807 Export, import, compare, and verify prompts and assistants across LangSmith environments Co-Authored-By: Claude Opus 5.5 --- .../export-import-prompts-and-assistants/design.md | 10 +++++----- .../export-import-prompts-and-assistants/proposal.md | 2 +- .../specs/prompt/spec.md | 2 +- .../export-import-prompts-and-assistants/tasks.md | 8 ++++---- 4 files changed, 11 insertions(+), 11 deletions(-) diff --git a/openspec/changes/export-import-prompts-and-assistants/design.md b/openspec/changes/export-import-prompts-and-assistants/design.md index a62c4fff3..6cd71ea8f 100644 --- a/openspec/changes/export-import-prompts-and-assistants/design.md +++ b/openspec/changes/export-import-prompts-and-assistants/design.md @@ -42,7 +42,7 @@ One prompt export looks like this: "format_version": 1, "source": { "workspace_id": "", "handle": "my-prompt", "commit_hash": "", "ref": "prod" }, "repo": { "handle": "my-prompt", "description": "...", "readme": "...", "tags": ["support"], "is_public": false }, - "commit": { "description": "...", "manifest": { "lc": 1, "type": "constructor", "id": ["langsmith", "playground", "PromptPlayground"], "kwargs": { "first": {}, "last": {} } }, "model_config": {}, "model_provider": "ChatOpenAI", "tags": ["prod"] }, + "commit": { "description": "...", "manifest": { "lc": 1, "type": "constructor", "id": ["langsmith", "playground", "PromptPlayground"], "kwargs": { "first": {}, "last": {} } }, "model_config": {}, "model_provider": "...", "tags": ["prod"] }, "requires": { "secrets": ["OPENAI_API_KEY"], "model_provider": "ChatOpenAI" } } ``` @@ -61,7 +61,7 @@ Import writes the destination's IDs to `.destinations.json`, keyed by ### Import changes an existing resource only with --update -A prompt that exists in the destination gets a new commit only with `--update`, and its parent is the destination's latest commit, so LangSmith keeps the earlier history. Import sends the file's commit description with each commit it creates. With `--update`, import also sets the destination repo's description, readme, tags and visibility to the file's values through `PATCH /api/v1/repos/{owner}/{repo}` when they differ. When only those repo fields differ and the manifest matches, import updates the repo and adds no commit. A commit tag in the file that the destination lacks, or that points at another commit there, also counts as a difference; with `--update`, import creates or moves the tag to the destination commit that matches the file, and adds no commit when the latest commit already matches. +A prompt that exists in the destination gets a new commit only with `--update`, and its parent is the destination's latest commit, so LangSmith keeps the earlier history. Import sends the file's commit description with each commit it creates. With `--update`, import also sets the destination repo's description, readme, tags and visibility to the file's values through `PATCH /api/v1/repos/{owner}/{repo}` when they differ. When only those repo fields differ and the manifest matches, import updates the repo and adds no commit. A commit tag in the file that the destination lacks, or that points at another commit there, also counts as a difference; with `--update`, import creates or moves the tag to the destination commit that matches the file, and adds no commit when the latest commit already matches. The tag calls name their commit by its `id`, which neither the commit read nor the repo read returns, so import finds a commit it did not create through the commit list, `GET /api/v1/commits/{owner}/{repo}`, whose entries carry both `id` and hash. The Agent Server keeps the name, graph ID and description an update leaves out, and merges the metadata an update sends into the metadata the assistant already has, as `AssistantPatch` in `reference/openapi/langchain/agent-server/openapi.json` documents. So with `--update`, import asks for each removal and then checks that it happened. It sends the file's name, graph ID, configuration and context in full, an empty object for a configuration or context the file lacks, and a null value for each metadata key the destination has and the file lacks. `AssistantPatch` documents no way to clear a description, so [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8) tries a null and an empty description against the live server, along with a null metadata value and a replaced configuration and context. Import then reads the assistant back, and any key or field the file lacks that is still there shows up as a difference, with exit status 2. The Agent Server saves each update as a new version of the assistant. @@ -73,13 +73,13 @@ Compare reads the destination, removes the fields the server sets (IDs, version ### Verify checks what the copy needs to run -Prompt verify looks up each name in `requires.secrets` among the workspace's secrets (`GET /api/v1/workspaces/current/secrets`, which `langstar secrets list` already calls). It also looks for a saved model configuration (`GET /api/v1/playground-settings`, which `langstar model-config list` calls) whose settings `id` ends in the class name that `requires.model_provider` holds, such as `ChatOpenAI`. The prompt runs without a saved model configuration, because its manifest carries the model, so verify reports a missing one as a note and leaves the exit status alone; [#807 Export, import, compare, and verify prompts and assistants across LangSmith environments](https://github.com/codekiln/langstar/issues/807) asks for the report, and Open Question 7 asks whether to keep it. Verify reports names only. Langstar finds secret names by walking the manifest for LangChain secret references, `{"lc": 1, "type": "secret", "id": ["OPENAI_API_KEY"]}`, the shape the Python SDK's `_manifest_has_secrets` matches. +Prompt verify looks up each name in `requires.secrets` among the workspace's secrets (`GET /api/v1/workspaces/current/secrets`, which `langstar secrets list` already calls). It also looks for a saved model configuration (`GET /api/v1/playground-settings`, which `langstar model-config list` calls) whose settings `id` ends in the class name that `requires.model_provider` holds, such as `ChatOpenAI`. Export takes that class name from the last element of the chat model constructor's `id` in the manifest, such as `["langchain", "chat_models", "openai", "ChatOpenAI"]`, because no source shows what LangSmith returns in `model_provider`; the file keeps that returned value as `commit.model_provider`, and task 1.2 records it. The prompt runs without a saved model configuration, because its manifest carries the model, so verify reports a missing one as a note and leaves the exit status alone; [#807 Export, import, compare, and verify prompts and assistants across LangSmith environments](https://github.com/codekiln/langstar/issues/807) asks for the report, and Open Question 7 asks whether to keep it. Verify prints the name of each missing secret and no secret value. Langstar finds secret names by walking the manifest for LangChain secret references, `{"lc": 1, "type": "secret", "id": ["OPENAI_API_KEY"]}`, the shape the Python SDK's `_manifest_has_secrets` matches. Assistant verify checks the graph with `POST /assistants/search` filtered to the graph ID; every graph on a deployment has a default assistant, so an empty result means the deployment does not serve the graph. It checks each prompt in `requires.prompts`, such as `support-system:prod`, by asking LangSmith for the commit that tag points at (`GET /api/v1/commits/{owner}/{repo}/{tag}`, the request `langstar prompt pull --commit prod` already sends). A 404 means the workspace lacks the prompt or its tag. ### Import runs verify's checks before it writes -Before it writes anything, import runs the checks verify runs against the destination, because [#807 Export, import, compare, and verify prompts and assistants across LangSmith environments](https://github.com/codekiln/langstar/issues/807) asks import to check the destination's dependencies. Prompt import stops when the destination workspace lacks a secret in `requires.secrets`, and assistant import stops when the deployment does not serve the file's graph or the workspace, from `--workspace-id` or the configured workspace, lacks a prompt in `requires.prompts`. Each names what is missing, writes nothing and exits with status 2, and `--dry-run` reports the same failed checks. A user therefore never gets a copy that cannot run, but has to create the destination's secrets and import the prompts before importing an assistant that refers to them; Open Question 8 asks whether import should write anyway and report the failed checks afterwards. +Before it writes anything, import runs the checks verify runs against the destination, because [#807 Export, import, compare, and verify prompts and assistants across LangSmith environments](https://github.com/codekiln/langstar/issues/807) asks import to check the destination's dependencies. Prompt import stops when the destination workspace lacks a secret in `requires.secrets`, and assistant import stops when the deployment does not serve the file's graph or the workspace, from `--workspace-id` or the configured workspace, lacks a prompt in `requires.prompts`. Each names what is missing, writes nothing and exits with status 2, and `--dry-run` reports the same failed checks. Import therefore writes a copy only when the destination has the secrets, graph and prompts that copy needs to run. In return, the user works in order: create the destination's secrets before importing a prompt, and import the prompts before importing an assistant that refers to them. Open Question 8 asks whether import should write anyway and report the failed checks afterwards. ### The user names an assistant's prompt references @@ -155,7 +155,7 @@ None. ### 7 - Should prompt verify report a missing saved model configuration? > Context from the drafting agent, for question 7. -> Issue #807 asks import to report missing model configurations, so the draft has prompt verify look for a saved model configuration (`GET /api/v1/playground-settings`) whose settings `id` ends in the prompt's `model_provider` class name, such as `ChatOpenAI`. Nothing fails at run time without one, because the prompt's manifest carries the model and its settings; the secret check is the one that finds a prompt that cannot authenticate. Reporting it as a failure would make verify exit with 2 for a prompt that runs. Recommendation: report a missing saved model configuration as a note that leaves the exit status alone. +> Issue #807 asks import to report missing model configurations, so the draft has prompt verify look for a saved model configuration (`GET /api/v1/playground-settings`) whose settings `id` ends in the class name the export file holds in `requires.model_provider`, such as `ChatOpenAI`. Nothing fails at run time without one, because the prompt's manifest carries the model and its settings; the secret check is the one that finds a prompt that cannot authenticate. Reporting it as a failure would make verify exit with 2 for a prompt that runs. Recommendation: report a missing saved model configuration as a note that leaves the exit status alone. diff --git a/openspec/changes/export-import-prompts-and-assistants/proposal.md b/openspec/changes/export-import-prompts-and-assistants/proposal.md index 4738febc7..f9a0e4f35 100644 --- a/openspec/changes/export-import-prompts-and-assistants/proposal.md +++ b/openspec/changes/export-import-prompts-and-assistants/proposal.md @@ -30,7 +30,7 @@ A prompt belongs to a workspace and an assistant belongs to a deployment, but bo ## Impact -- SDK: `sdk/src/prompts.rs` gains a commit read that asks for the model, a commit description on commit create, the new commit's `id`, a repo update, and calls that list, create and move commit tags. `sdk/src/assistants.rs` gains the `context` and `version` fields, `assistant_id` and `if_exists` on create, `graph_id` on update, and a search by graph ID. `sdk/src/deployments.rs` gains the deployment's `url`. +- SDK: `sdk/src/prompts.rs` gains a commit read that asks for the model, a commit description on commit create, the new commit's `id`, a repo update, a commit list that returns each commit's `id`, and calls that list, create and move commit tags. `sdk/src/assistants.rs` gains the `context` and `version` fields, `assistant_id` and `if_exists` on create, `graph_id` on update, and a search by graph ID. `sdk/src/deployments.rs` gains the deployment's `url`. - CLI: new subcommands in `cli/src/commands/prompt.rs` and `cli/src/commands/assistant.rs`, and a shared module for the export file, the destination record, comparison and the exit status. `resolve_deployment_url` in `cli/src/deployment_utils.rs` also returns the Agent Server URL of a deployment built from GitHub, which every `langstar assistant` command needs. - Tests need a second workspace, a second deployment and one model provider key for the structured output check. codekiln creates the workspace and the key, and langstar's tests read them through `fnox.toml` and the CI workflow; the `TestDeploymentConfig` fixture creates the deployment on the first run and the tests reuse it after that. - Docs: a new usage page on moving prompts and assistants between environments. diff --git a/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md index 5d1e6dd66..8c61d35b2 100644 --- a/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md +++ b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md @@ -45,7 +45,7 @@ Exporting the same commit twice SHALL produce identical files. The file SHALL ho ### Requirement: List the credentials a prompt needs without their values -The export file SHALL list, under `requires.secrets`, the name of every secret the manifest refers to, and under `requires.model_provider`, the model's provider class name as LangSmith returns it, such as `ChatOpenAI`. It SHALL hold no secret values. +The export file SHALL list, under `requires.secrets`, the name of every secret the manifest refers to, and under `requires.model_provider`, the class name at the end of the chat model constructor's `id` in the manifest, such as `ChatOpenAI`. It SHALL hold no secret values. #### Scenario: Model reads an API key secret diff --git a/openspec/changes/export-import-prompts-and-assistants/tasks.md b/openspec/changes/export-import-prompts-and-assistants/tasks.md index 125937c08..5c84f54d6 100644 --- a/openspec/changes/export-import-prompts-and-assistants/tasks.md +++ b/openspec/changes/export-import-prompts-and-assistants/tasks.md @@ -5,7 +5,7 @@ Each numbered group is one pull request, in the order listed. ## 1. Check the API round trip before building on it - [ ] 1.1 Extract the commit get, commit create, repo update (`PATCH /api/v1/repos/{owner}/{repo}`), repo tags (`/api/v1/repos/{owner}/{repo}/tags` and `/api/v1/repos/{owner}/{repo}/tags/{tag_name}`), workspace secrets and playground settings endpoints from `reference/openapi/langchain/langsmith/openapi.json` into `reference/api-specs/langsmith/` and add their jq queries to `FRAGMENTS.md`; verify by rerunning each query and getting the same file -- [ ] 1.2 Write `reference/research/807-openapi-validation.md`, recording from the live API: the manifest returned with `include=model` for a structured prompt with a model and the outer `id` of that manifest, whether pushing that manifest to a second workspace returns the same model, what pushing an identical manifest returns, what the Agent Server returns for `context`, whether an Agent Server assistant update that sends a metadata key with a null value removes that key, whether an update that sends `config` or `context` replaces the assistant's object or merges into it, whether an update can clear an assistant's description, and what `POST /api/v1/prompts/invoke_prompt` returns for a prompt with a model; verify that every claim in the report names the request that produced it +- [ ] 1.2 Write `reference/research/807-openapi-validation.md`, recording from the live API: the manifest returned with `include=model` for a structured prompt with a model, the outer `id` of that manifest and the `model_provider` value returned beside it, whether pushing that manifest to a second workspace returns the same model, what pushing an identical manifest returns, what the Agent Server returns for `context`, whether an Agent Server assistant update that sends a metadata key with a null value removes that key, whether an update that sends `config` or `context` replaces the assistant's object or merges into it, whether an update can clear an assistant's description, and what `POST /api/v1/prompts/invoke_prompt` returns for a prompt with a model; verify that every claim in the report names the request that produced it ## 2. Test environments @@ -16,7 +16,7 @@ Each numbered group is one pull request, in the order listed. ## 3. SDK reads and writes the fields export and import need - [ ] 3.1 Add a commit read that sends `include=model` and returns `description`, `model_config` and `model_provider` to `sdk/src/prompts.rs`; verify with an httpmock test that asserts the query string and the parsed fields, and an integration test that reads a prompt with a model -- [ ] 3.2 Add the commit's `id` to `CommitData` in `sdk/src/prompts.rs`, and add calls that list, create and move commit tags, which name their commit by that `id` in `POST /api/v1/repos/{owner}/{repo}/tags` and `PATCH /api/v1/repos/{owner}/{repo}/tags/{tag_name}`; verify with httpmock tests that assert `commit_id` in each request body, and an integration test that pushes two commits, tags the first, moves the tag to the second, and reads the tag back through the SDK +- [ ] 3.2 Add the commit's `id` to `CommitData` in `sdk/src/prompts.rs`, a call that lists a prompt's commits with their `id` and hash (`GET /api/v1/commits/{owner}/{repo}`), and calls that list, create and move commit tags, which name their commit by that `id` in `POST /api/v1/repos/{owner}/{repo}/tags` and `PATCH /api/v1/repos/{owner}/{repo}/tags/{tag_name}`; verify with httpmock tests that assert `commit_id` in each request body, and an integration test that pushes two commits, tags the first, finds the second commit's `id` through the commit list, moves the tag to it, and reads the tag back through the SDK - [ ] 3.3 Add `context` and `version` to `Assistant`, `context` and `graph_id` to `UpdateAssistantRequest`, `context` to `CreateAssistantRequest`, `assistant_id` and `if_exists` to `CreateAssistantRequest`, and a `graph_id` filter to `AssistantSearchRequest`, in `sdk/src/assistants.rs`; verify with httpmock tests of the request bodies and an integration test that creates an assistant with context, reads it back with its version, finds it by graph ID, and updates its graph ID and reads the new graph ID back - [ ] 3.4 Add `description` to `CommitRequest`, and a repo update that sends `PATCH /api/v1/repos/{owner}/{repo}` with `description`, `readme`, `tags` and `is_public`, to `sdk/src/prompts.rs`; verify with httpmock tests of the request bodies and an integration test that pushes a commit with a description, updates the repo's description, readme, tags and visibility, and reads each back @@ -29,7 +29,7 @@ Each numbered group is one pull request, in the order listed. ## 5. prompt export -- [ ] 5.1 Add `langstar prompt export` with `--commit`, `--output` and the scoping flags; verify with integration tests that create prompts through the SDK (a `json_schema` prompt, a `function_calling` prompt with a nested schema, a prompt with several messages and a placeholder, and a prompt with a model and metadata), export each through the CLI, and assert every field of the file against the SDK's read of the commit +- [ ] 5.1 Add `langstar prompt export` with `--commit`, `--output` and the scoping flags; verify with integration tests that create prompts through the SDK (a `json_schema` prompt, a `function_calling` prompt with a nested schema, a prompt with several messages and a placeholder, and a prompt with a model and metadata), export each through the CLI, and assert every field of the file against the SDK's read of the commit, and an export with `--commit prod` of a prompt whose `prod` tag points at an earlier commit, asserting that `source.commit_hash` and the manifest are that earlier commit's - [ ] 5.2 Document `prompt export` in `docs/usage/` and the README; verify that each documented command runs as written ## 6. prompt compare and verify @@ -59,7 +59,7 @@ Each numbered group is one pull request, in the order listed. ## 10. assistant import - [ ] 10.1 Add `langstar assistant import` with `--workspace-id`, the graph check and prompt check of `assistant verify` from 9.2 run before it writes, creation, read-back, the destination record and `--record`; verify with an integration test that creates a tagged prompt in the test workspace, imports an assistant that refers to it into the destination deployment and checks name, graph ID, configuration, context, metadata and prompt references through the Agent Server API, that `--record ` writes the destination record to that path, and that a file whose graph ID the deployment lacks and a file with a missing prompt tag each create no assistant and exit with status 2 -- [ ] 10.2 Add `--update`, `--assistant-id` and `--dry-run`; verify with integration tests that a repeated import creates no assistant, that `--update` changes the recorded one, its graph ID among the fields, that `--update` of an assistant with metadata `{"kept": 1, "stale": 2}` from a file with `{"kept": 1}` leaves `{"kept": 1}` or reports `/metadata/stale` and exits with status 2, whichever task 1.2 found the server does, that `--update` of an assistant with a description and context from a file with neither clears both or reports `/description` and `/context` and exits with status 2, whichever task 1.2 found the server does, that a deleted recorded assistant exits with status 1, and that `--dry-run` creates no assistant, changes none and leaves the destination record as it was +- [ ] 10.2 Add `--update`, `--assistant-id` and `--dry-run`; verify with integration tests that a repeated import creates no assistant, that `--update` changes the recorded one, its graph ID among the fields, that `--update --assistant-id ` naming an assistant other than the recorded one updates that assistant, leaves the recorded one unchanged and writes the new ID to the destination record, that `--update` of an assistant with metadata `{"kept": 1, "stale": 2}` from a file with `{"kept": 1}` leaves `{"kept": 1}` or reports `/metadata/stale` and exits with status 2, whichever task 1.2 found the server does, that `--update` of an assistant with a description and context from a file with neither clears both or reports `/description` and `/context` and exits with status 2, whichever task 1.2 found the server does, that a deleted recorded assistant exits with status 1, and that `--dry-run` creates no assistant, changes none and leaves the destination record as it was - [ ] 10.3 Add an end-to-end test that imports prompts into the destination workspace and then an assistant that refers to them onto the destination deployment, and runs `assistant verify`; verify that it passes and that it fails after deleting one prompt tag - [ ] 10.4 Document assistant import, the deployment scope of assistants, and how export and import relate to the `create`, `get` and `update` commands of [#728 Implement CRUD command structure (create, get, update) for prompt commands](https://github.com/codekiln/langstar/issues/728); verify that each documented command runs as written From 752e5a9442b861c58f0c7ee25f5dab9cc9779f26 Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Tue, 13 Oct 2026 19:16:46 -0400 Subject: [PATCH 09/90] =?UTF-8?q?=F0=9F=93=9A=20docs(openspec):=20exit=20w?= =?UTF-8?q?ith=203=20for=20a=20difference,=20leave=20clap's=202=20for=20us?= =?UTF-8?q?age=20errors,=20and=20drop=20unused=20create=20fields?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit clap exits with 2 on a usage error, so compare, verify and import now exit with 3 for a difference or a failed check and 1 for any other error. prompt import checks --workspace-id itself and exits with 1. Task 3.3 drops assistant_id and if_exists, task 1.1 extracts the commit list, and tasks 5.1 and 8.1 test a missing prompt and assistant. #807 Export, import, compare, and verify prompts and assistants across LangSmith environments Co-Authored-By: Claude Opus 5.5 --- .../design.md | 24 ++++++------ .../proposal.md | 4 +- .../specs/assistant/spec.md | 24 ++++++------ .../specs/prompt/spec.md | 37 +++++++++++-------- .../tasks.md | 22 +++++------ 5 files changed, 58 insertions(+), 53 deletions(-) diff --git a/openspec/changes/export-import-prompts-and-assistants/design.md b/openspec/changes/export-import-prompts-and-assistants/design.md index 6cd71ea8f..40ea4b005 100644 --- a/openspec/changes/export-import-prompts-and-assistants/design.md +++ b/openspec/changes/export-import-prompts-and-assistants/design.md @@ -4,9 +4,9 @@ `langstar prompt push` in `cli/src/commands/prompt.rs` builds one shape of prompt: a single human message, with an optional schema from [#402 structured output prompts](https://github.com/codekiln/langstar/issues/402). `langstar prompt pull` prints any manifest LangSmith returns, but `PromptClient` in `sdk/src/prompts.rs` never asks LangSmith for the model, so that manifest has none. `PromptClient` keeps the manifest as raw JSON and has a typed `StructuredPrompt` view that holds messages, the schema and the method. -`langstar assistant` in `cli/src/commands/assistant.rs` takes `--deployment` on every subcommand and sends requests to that deployment's Agent Server. `AssistantClient` in `sdk/src/assistants.rs` has no `context` or `version` field, its create request cannot set `assistant_id` or `if_exists`, and its search cannot filter by graph ID. +`langstar assistant` in `cli/src/commands/assistant.rs` takes `--deployment` on every subcommand and sends requests to that deployment's Agent Server. `AssistantClient` in `sdk/src/assistants.rs` has no `context` or `version` field and its search cannot filter by graph ID. -Langstar reads one API key and one workspace at a time, from flags, environment variables or the config file. Every error exits with status 1. +Langstar reads one API key and one workspace at a time, from flags, environment variables or the config file. A langstar error exits with status 1, and a usage error that clap rejects, such as a mistyped flag, exits with status 2. ## Goals / Non-Goals @@ -61,11 +61,11 @@ Import writes the destination's IDs to `.destinations.json`, keyed by ### Import changes an existing resource only with --update -A prompt that exists in the destination gets a new commit only with `--update`, and its parent is the destination's latest commit, so LangSmith keeps the earlier history. Import sends the file's commit description with each commit it creates. With `--update`, import also sets the destination repo's description, readme, tags and visibility to the file's values through `PATCH /api/v1/repos/{owner}/{repo}` when they differ. When only those repo fields differ and the manifest matches, import updates the repo and adds no commit. A commit tag in the file that the destination lacks, or that points at another commit there, also counts as a difference; with `--update`, import creates or moves the tag to the destination commit that matches the file, and adds no commit when the latest commit already matches. The tag calls name their commit by its `id`, which neither the commit read nor the repo read returns, so import finds a commit it did not create through the commit list, `GET /api/v1/commits/{owner}/{repo}`, whose entries carry both `id` and hash. +A prompt that exists in the destination gets a new commit only with `--update`, and its parent is the destination's latest commit, so LangSmith keeps the earlier history. Import sends the file's commit description with each commit it creates. With `--update`, import also sets the destination repo's description, readme, tags and visibility to the file's values through `PATCH /api/v1/repos/{owner}/{repo}` when they differ. When only those repo fields differ and the manifest matches, import updates the repo and adds no commit. A commit tag in the file that the destination lacks, or that points at another commit there, also counts as a difference; with `--update`, import creates or moves the tag to the destination commit that matches the file, and adds no commit when the latest commit already matches. The tag endpoints take a commit's `id`, which differs from its hash. The commit read and the repo read return no `id`, so import looks the commit up in the commit list, `GET /api/v1/commits/{owner}/{repo}`, whose entries carry both the `id` and the hash. -The Agent Server keeps the name, graph ID and description an update leaves out, and merges the metadata an update sends into the metadata the assistant already has, as `AssistantPatch` in `reference/openapi/langchain/agent-server/openapi.json` documents. So with `--update`, import asks for each removal and then checks that it happened. It sends the file's name, graph ID, configuration and context in full, an empty object for a configuration or context the file lacks, and a null value for each metadata key the destination has and the file lacks. `AssistantPatch` documents no way to clear a description, so [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8) tries a null and an empty description against the live server, along with a null metadata value and a replaced configuration and context. Import then reads the assistant back, and any key or field the file lacks that is still there shows up as a difference, with exit status 2. The Agent Server saves each update as a new version of the assistant. +The Agent Server keeps the name, graph ID and description an update leaves out, and merges the metadata an update sends into the metadata the assistant already has, as `AssistantPatch` in `reference/openapi/langchain/agent-server/openapi.json` documents. So with `--update`, import asks for each removal and then checks that it happened. It sends the file's name, graph ID, configuration and context in full, an empty object for a configuration or context the file lacks, and a null value for each metadata key the destination has and the file lacks. `AssistantPatch` documents no way to clear a description, so [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8) tries a null and an empty description against the live server, along with a null metadata value and a replaced configuration and context. Import then reads the assistant back, and any key or field the file lacks that is still there shows up as a difference, with exit status 3. The Agent Server saves each update as a new version of the assistant. -Without `--update`, import lists the differences and exits with status 2. `--dry-run` reports the action import would take and writes nothing. +Without `--update`, import lists the differences and exits with status 3. `--dry-run` reports the action import would take and writes nothing. ### Compare reports differences by JSON path @@ -73,21 +73,21 @@ Compare reads the destination, removes the fields the server sets (IDs, version ### Verify checks what the copy needs to run -Prompt verify looks up each name in `requires.secrets` among the workspace's secrets (`GET /api/v1/workspaces/current/secrets`, which `langstar secrets list` already calls). It also looks for a saved model configuration (`GET /api/v1/playground-settings`, which `langstar model-config list` calls) whose settings `id` ends in the class name that `requires.model_provider` holds, such as `ChatOpenAI`. Export takes that class name from the last element of the chat model constructor's `id` in the manifest, such as `["langchain", "chat_models", "openai", "ChatOpenAI"]`, because no source shows what LangSmith returns in `model_provider`; the file keeps that returned value as `commit.model_provider`, and task 1.2 records it. The prompt runs without a saved model configuration, because its manifest carries the model, so verify reports a missing one as a note and leaves the exit status alone; [#807 Export, import, compare, and verify prompts and assistants across LangSmith environments](https://github.com/codekiln/langstar/issues/807) asks for the report, and Open Question 7 asks whether to keep it. Verify prints the name of each missing secret and no secret value. Langstar finds secret names by walking the manifest for LangChain secret references, `{"lc": 1, "type": "secret", "id": ["OPENAI_API_KEY"]}`, the shape the Python SDK's `_manifest_has_secrets` matches. +Prompt verify looks up each name in `requires.secrets` among the workspace's secrets (`GET /api/v1/workspaces/current/secrets`, which `langstar secrets list` already calls). It also looks for a saved model configuration (`GET /api/v1/playground-settings`, which `langstar model-config list` calls) whose settings `id` ends in the class name that `requires.model_provider` holds, such as `ChatOpenAI`. Export reads that class name from the last element of the chat model constructor's `id` in the manifest, such as `["langchain", "chat_models", "openai", "ChatOpenAI"]`. LangSmith also returns a `model_provider` field beside the manifest, and the file keeps it as `commit.model_provider`. No SDK or spec shows what that field holds, so [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8) records it. The prompt runs without a saved model configuration, because its manifest carries the model, so verify reports a missing one as a note and leaves the exit status alone; [#807 Export, import, compare, and verify prompts and assistants across LangSmith environments](https://github.com/codekiln/langstar/issues/807) asks for the report, and Open Question 7 asks whether to keep it. Verify prints the name of each missing secret and no secret value. Langstar finds secret names by walking the manifest for LangChain secret references, `{"lc": 1, "type": "secret", "id": ["OPENAI_API_KEY"]}`, the shape the Python SDK's `_manifest_has_secrets` matches. Assistant verify checks the graph with `POST /assistants/search` filtered to the graph ID; every graph on a deployment has a default assistant, so an empty result means the deployment does not serve the graph. It checks each prompt in `requires.prompts`, such as `support-system:prod`, by asking LangSmith for the commit that tag points at (`GET /api/v1/commits/{owner}/{repo}/{tag}`, the request `langstar prompt pull --commit prod` already sends). A 404 means the workspace lacks the prompt or its tag. ### Import runs verify's checks before it writes -Before it writes anything, import runs the checks verify runs against the destination, because [#807 Export, import, compare, and verify prompts and assistants across LangSmith environments](https://github.com/codekiln/langstar/issues/807) asks import to check the destination's dependencies. Prompt import stops when the destination workspace lacks a secret in `requires.secrets`, and assistant import stops when the deployment does not serve the file's graph or the workspace, from `--workspace-id` or the configured workspace, lacks a prompt in `requires.prompts`. Each names what is missing, writes nothing and exits with status 2, and `--dry-run` reports the same failed checks. Import therefore writes a copy only when the destination has the secrets, graph and prompts that copy needs to run. In return, the user works in order: create the destination's secrets before importing a prompt, and import the prompts before importing an assistant that refers to them. Open Question 8 asks whether import should write anyway and report the failed checks afterwards. +Before it writes anything, import runs the checks verify runs against the destination, because [#807 Export, import, compare, and verify prompts and assistants across LangSmith environments](https://github.com/codekiln/langstar/issues/807) asks import to check the destination's dependencies. Prompt import stops when the destination workspace lacks a secret in `requires.secrets`, and assistant import stops when the deployment does not serve the file's graph or the workspace, from `--workspace-id` or the configured workspace, lacks a prompt in `requires.prompts`. Each names what is missing, writes nothing and exits with status 3, and `--dry-run` reports the same failed checks. Import therefore writes a copy only when the destination has the secrets, graph and prompts that copy needs to run. In return, the user works in order: create the destination's secrets before importing a prompt, and import the prompts before importing an assistant that refers to them. Open Question 8 asks whether import should write anyway and report the failed checks afterwards. ### The user names an assistant's prompt references An assistant refers to a prompt through whatever configuration key its graph reads, so langstar cannot find the references by itself. `assistant export --prompt-ref ` names each one. The assistant's configuration should name each prompt by handle and tag, such as `support-system:prod`, because the same prompt gets a different commit hash in each workspace. Import copies the commit tags to the destination workspace, so when the graph on the destination deployment asks LangSmith for `support-system:prod`, it gets the same prompt. -### compare, verify and import exit with 2 when they find a difference +### compare, verify and import exit with 3 when they find a difference -Status 0 means the resource matches and every check passed. Status 2 means langstar found a difference or a failed check. Status 1 stays the status for every other error, as in every other langstar command. `diff` uses 1 for differences and 2 for errors; langstar keeps 1 for errors so that a script testing for failure sees the same status from every command. +Status 0 means the resource matches and every check passed. Status 3 means langstar found a difference or a failed check. Status 1 means any other error. Status 2 stays clap's: `Cli::parse()` in `run` in `cli/src/main.rs` exits with 2 when a user mistypes a flag or leaves out a required one, and this change leaves that alone, so a script never reads a typo as a difference. `prompt import` declares `--workspace-id` optional, as the other `langstar prompt` subcommands do, and checks for it itself, so a missing flag exits with status 1 and an error that names the flag. `diff` uses 1 for differences and 2 for errors; langstar keeps 1 for errors so that a script testing for failure sees the same status from every command. ### A Python script checks that an imported structured prompt still returns structured data @@ -113,7 +113,7 @@ None. ### 1 - Should compare, verify and import exit with 2 when they find a difference? > Context from the drafting agent, for question 1. -> The draft uses 0 for a match, 2 for a difference or failed check, and 1 for every other error, as in every other langstar command. `diff` and `git diff --exit-code` use 1 for differences, which would make a difference look like an error to scripts written against langstar. Recommendation: keep 0, 2 and 1. +> Status 2 is taken: clap exits with 2 when a user mistypes a flag or leaves out a required one, so a difference reported as 2 would look like a typo to a script. The draft uses 0 for a match, 3 for a difference or failed check and 1 for every other error, and leaves clap's 2 alone. The other choice makes `run` in `cli/src/main.rs` exit with 1 on a usage error, which frees 2 for differences but changes the status every langstar command gives for a mistyped flag. `diff` and `git diff --exit-code` use 1 for differences, which would make a difference look like an error. Recommendation: 3 for a difference. @@ -155,13 +155,13 @@ None. ### 7 - Should prompt verify report a missing saved model configuration? > Context from the drafting agent, for question 7. -> Issue #807 asks import to report missing model configurations, so the draft has prompt verify look for a saved model configuration (`GET /api/v1/playground-settings`) whose settings `id` ends in the class name the export file holds in `requires.model_provider`, such as `ChatOpenAI`. Nothing fails at run time without one, because the prompt's manifest carries the model and its settings; the secret check is the one that finds a prompt that cannot authenticate. Reporting it as a failure would make verify exit with 2 for a prompt that runs. Recommendation: report a missing saved model configuration as a note that leaves the exit status alone. +> Issue #807 asks import to report missing model configurations, so the draft has prompt verify look for a saved model configuration (`GET /api/v1/playground-settings`) whose settings `id` ends in the class name the export file holds in `requires.model_provider`, such as `ChatOpenAI`. Nothing fails at run time without one, because the prompt's manifest carries the model and its settings; the secret check is the one that finds a prompt that cannot authenticate. Reporting it as a failure would make verify exit with 3 for a prompt that runs. Recommendation: report a missing saved model configuration as a note that leaves the exit status alone. ### 8 - Should a failed dependency check stop import before it writes? > Context from the drafting agent, for question 8. -> Issue #807 asks import to check the destination's dependencies: the secrets a prompt's model reads, and an assistant's graph and referenced prompts. The draft runs those checks before import writes and writes nothing when one fails, so a user never has to find and delete a copy that cannot run. The cost is order: the destination workspace needs its secrets before a prompt import, and its prompts before an assistant import. The other choice writes the copy anyway and reports the failed checks with status 2 afterwards, which lets a user import in any order and fix the destination later. Recommendation: stop before writing. +> Issue #807 asks import to check the destination's dependencies: the secrets a prompt's model reads, and an assistant's graph and referenced prompts. The draft runs those checks before import writes and writes nothing when one fails, so a user never has to find and delete a copy that cannot run. The cost is order: the destination workspace needs its secrets before a prompt import, and its prompts before an assistant import. The other choice writes the copy anyway and reports the failed checks with status 3 afterwards, which lets a user import in any order and fix the destination later. Recommendation: stop before writing. diff --git a/openspec/changes/export-import-prompts-and-assistants/proposal.md b/openspec/changes/export-import-prompts-and-assistants/proposal.md index f9a0e4f35..db8a7a789 100644 --- a/openspec/changes/export-import-prompts-and-assistants/proposal.md +++ b/openspec/changes/export-import-prompts-and-assistants/proposal.md @@ -16,7 +16,7 @@ A prompt belongs to a workspace and an assistant belongs to a deployment, but bo - `langstar assistant export`, `import`, `compare` and `verify` do the same for assistants on a deployment. The file holds the name, description, graph ID, configuration, context, metadata and any prompt names the assistant refers to. `verify` checks that the destination deployment has the graph and that the destination workspace has the referenced prompts. - Import records each destination ID (the new commit hash, or the new assistant ID) in a separate file next to the export, so importing the same file again finds the copy it made earlier instead of creating another one. - Import runs the checks of `verify` before it writes, and writes nothing when one fails. -- `compare`, `verify` and `import` share one exit status scheme and print a JSON report with `-f json`. +- `compare`, `verify` and `import` share one exit status scheme, 0 for a match, 3 for a difference or a failed check and 1 for any other error, and print a JSON report with `-f json`. Status 2 stays clap's usage error. ## Capabilities @@ -30,7 +30,7 @@ A prompt belongs to a workspace and an assistant belongs to a deployment, but bo ## Impact -- SDK: `sdk/src/prompts.rs` gains a commit read that asks for the model, a commit description on commit create, the new commit's `id`, a repo update, a commit list that returns each commit's `id`, and calls that list, create and move commit tags. `sdk/src/assistants.rs` gains the `context` and `version` fields, `assistant_id` and `if_exists` on create, `graph_id` on update, and a search by graph ID. `sdk/src/deployments.rs` gains the deployment's `url`. +- SDK: `sdk/src/prompts.rs` gains a commit read that asks for the model, a commit description on commit create, the new commit's `id`, a repo update, a commit list that returns each commit's `id`, and calls that list, create and move commit tags. `sdk/src/assistants.rs` gains the `context` and `version` fields, `graph_id` on update, and a search by graph ID. `sdk/src/deployments.rs` gains the deployment's `url`. - CLI: new subcommands in `cli/src/commands/prompt.rs` and `cli/src/commands/assistant.rs`, and a shared module for the export file, the destination record, comparison and the exit status. `resolve_deployment_url` in `cli/src/deployment_utils.rs` also returns the Agent Server URL of a deployment built from GitHub, which every `langstar assistant` command needs. - Tests need a second workspace, a second deployment and one model provider key for the structured output check. codekiln creates the workspace and the key, and langstar's tests read them through `fnox.toml` and the CI workflow; the `TestDeploymentConfig` fixture creates the deployment on the first run and the tests reuse it after that. - Docs: a new usage page on moving prompts and assistants between environments. diff --git a/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md b/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md index 33dfaa13e..53e674e19 100644 --- a/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md +++ b/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md @@ -71,18 +71,18 @@ Every `langstar assistant` subcommand SHALL take a required `--deployment` flag ### Requirement: Check the destination before importing an assistant -Before it writes, `langstar assistant import` SHALL run the graph check and the prompt check of `assistant verify`, reading prompts from `--workspace-id` or the configured workspace. When a check fails it SHALL write nothing, name each missing graph and prompt, and exit with status 2. +Before it writes, `langstar assistant import` SHALL run the graph check and the prompt check of `assistant verify`, reading prompts from `--workspace-id` or the configured workspace. When a check fails it SHALL write nothing, name each missing graph and prompt, and exit with status 3. #### Scenario: Referenced prompt missing at import - **WHEN** the file lists `support-system:prod` under `requires.prompts` - **AND** the workspace has no `support-system` prompt with a `prod` tag - **AND** a user runs `langstar assistant import support-bot.json --deployment my-agent-prod` -- **THEN** langstar creates no assistant, reports `support-system:prod` as missing and exits with status 2 +- **THEN** langstar creates no assistant, reports `support-system:prod` as missing and exits with status 3 ### Requirement: Find the earlier copy instead of creating another -When the destination record holds an assistant ID for the deployment, `langstar assistant import` SHALL compare the file with that assistant and create nothing. It SHALL exit with status 0 when they match, and list the differences and exit with status 2 when they differ, unless the user passes `--update`. When the recorded assistant no longer exists, it SHALL exit with status 1 and say so. +When the destination record holds an assistant ID for the deployment, `langstar assistant import` SHALL compare the file with that assistant and create nothing. It SHALL exit with status 0 when they match, and list the differences and exit with status 3 when they differ, unless the user passes `--update`. When the recorded assistant no longer exists, it SHALL exit with status 1 and say so. #### Scenario: Import the same file again @@ -111,7 +111,7 @@ When the destination record holds an assistant ID for the deployment, `langstar ### Requirement: Remove on update what the file lacks -`langstar assistant import --update` SHALL remove each metadata key the destination assistant has and the file lacks, and clear a description or context the file lacks. When such a key or field remains after the update, it SHALL list it as a difference and exit with status 2. +`langstar assistant import --update` SHALL remove each metadata key the destination assistant has and the file lacks, and clear a description or context the file lacks. When such a key or field remains after the update, it SHALL list it as a difference and exit with status 3. #### Scenario: Remove a metadata key the file lacks @@ -124,13 +124,13 @@ When the destination record holds an assistant ID for the deployment, `langstar #### Scenario: Deployment keeps a removed metadata key - **WHEN** the Agent Server keeps `stale` in the assistant's metadata after the update -- **THEN** langstar lists `/metadata/stale` as a difference and exits with status 2 +- **THEN** langstar lists `/metadata/stale` as a difference and exits with status 3 #### Scenario: Deployment keeps a description the file lacks - **WHEN** the recorded assistant has a description and the file has none - **AND** the Agent Server keeps that description after the update -- **THEN** langstar lists `/description` as a difference and exits with status 2 +- **THEN** langstar lists `/description` as a difference and exits with status 3 ### Requirement: Dry run an assistant import @@ -143,12 +143,12 @@ When the destination record holds an assistant ID for the deployment, `langstar ### Requirement: Read the assistant back after import -After writing, `langstar assistant import` SHALL read the assistant back from the deployment and compare it with the file. When they differ it SHALL list the differences and exit with status 2. +After writing, `langstar assistant import` SHALL read the assistant back from the deployment and compare it with the file. When they differ it SHALL list the differences and exit with status 3. #### Scenario: Deployment drops a configuration key - **WHEN** the assistant read back lacks a configuration key the file holds -- **THEN** langstar lists that key as a difference and exits with status 2 +- **THEN** langstar lists that key as a difference and exits with status 3 ### Requirement: Compare an assistant with its export file @@ -157,22 +157,22 @@ After writing, `langstar assistant import` SHALL read the assistant back from th #### Scenario: Metadata changed on the deployment - **WHEN** someone changed a metadata value on the deployment's assistant -- **THEN** langstar lists that metadata path with both values and exits with status 2 +- **THEN** langstar lists that metadata path with both values and exits with status 3 ### Requirement: Verify that an assistant can run on a deployment -`langstar assistant verify --deployment ` SHALL do what `compare` does and also check that the deployment serves the file's graph ID and that the workspace, from `--workspace-id` or the configured workspace, has each prompt under `requires.prompts`. It SHALL name each missing graph and prompt and exit with status 2 when any check fails. +`langstar assistant verify --deployment ` SHALL do what `compare` does and also check that the deployment serves the file's graph ID and that the workspace, from `--workspace-id` or the configured workspace, has each prompt under `requires.prompts`. It SHALL name each missing graph and prompt and exit with status 3 when any check fails. #### Scenario: Graph missing on the destination - **WHEN** the destination deployment serves no graph with the file's graph ID -- **THEN** langstar reports the missing graph ID and exits with status 2 +- **THEN** langstar reports the missing graph ID and exits with status 3 #### Scenario: Referenced prompt missing - **WHEN** the file lists `support-system:prod` under `requires.prompts` - **AND** the workspace has no `support-system` prompt with a `prod` tag -- **THEN** langstar reports `support-system:prod` as missing and exits with status 2 +- **THEN** langstar reports `support-system:prod` as missing and exits with status 3 ### Requirement: Assistant compare, verify and import use the prompt commands' exit statuses and JSON report diff --git a/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md index 8c61d35b2..dfb4b2781 100644 --- a/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md +++ b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md @@ -55,24 +55,24 @@ The export file SHALL list, under `requires.secrets`, the name of every secret t ### Requirement: Import names its destination workspace -`langstar prompt import ` SHALL require `--workspace-id`. It SHALL refuse to run when the flag is missing, even when `LANGSMITH_WORKSPACE_ID` is set. +`langstar prompt import ` SHALL require `--workspace-id`. It SHALL refuse to run when the flag is missing, even when `LANGSMITH_WORKSPACE_ID` is set, and exit with status 1 and an error naming the flag. #### Scenario: Missing destination workspace - **WHEN** a user runs `langstar prompt import my-prompt.json` without `--workspace-id` - **AND** `LANGSMITH_WORKSPACE_ID` is set -- **THEN** langstar exits with status 1 and an error asking for `--workspace-id` +- **THEN** langstar exits with status 1 and an error naming `--workspace-id` ### Requirement: Check the destination before importing a prompt -Before it writes, `langstar prompt import` SHALL run the checks of `prompt verify` against the destination workspace. When the workspace lacks a secret in `requires.secrets` it SHALL write nothing, name each missing secret and exit with status 2. A missing saved model configuration SHALL be a note, as in `prompt verify`. +Before it writes, `langstar prompt import` SHALL run the checks of `prompt verify` against the destination workspace. When the workspace lacks a secret in `requires.secrets` it SHALL write nothing, name each missing secret and exit with status 3. A missing saved model configuration SHALL be a note, as in `prompt verify`. #### Scenario: Secret missing at import - **WHEN** the file lists `OPENAI_API_KEY` under `requires.secrets` - **AND** the destination workspace has no secret named `OPENAI_API_KEY` - **AND** a user runs `langstar prompt import my-prompt.json --workspace-id ` -- **THEN** langstar creates no prompt, reports `OPENAI_API_KEY` as missing and exits with status 2 +- **THEN** langstar creates no prompt, reports `OPENAI_API_KEY` as missing and exits with status 3 ### Requirement: Import a prompt that does not exist yet @@ -87,13 +87,13 @@ When the destination workspace has no prompt with the file's handle, `langstar p ### Requirement: Change an existing prompt only with --update -When the destination prompt exists and its latest commit differs from the file, `langstar prompt import` SHALL change nothing and exit with status 2, listing the differences, unless the user passes `--update`. With `--update` it SHALL push the file's manifest and commit description as a new commit whose parent is the destination's latest commit, and move the file's tags to it. +When the destination prompt exists and its latest commit differs from the file, `langstar prompt import` SHALL change nothing and exit with status 3, listing the differences, unless the user passes `--update`. With `--update` it SHALL push the file's manifest and commit description as a new commit whose parent is the destination's latest commit, and move the file's tags to it. #### Scenario: Destination differs and --update is missing - **WHEN** the destination prompt's latest commit differs from the file - **AND** the user runs `langstar prompt import` without `--update` -- **THEN** langstar pushes nothing, lists the differences and exits with status 2 +- **THEN** langstar pushes nothing, lists the differences and exits with status 3 #### Scenario: Update an existing prompt @@ -120,7 +120,7 @@ When the destination repo's description, readme, tags or visibility differ from ### Requirement: Move commit tags only with --update -When a commit tag in the file is missing in the destination or points at a different commit, `langstar prompt import` SHALL list the tag as a difference and exit with status 2. With `--update` it SHALL create or move the tag to the destination commit that matches the file, and add no commit when the latest commit already matches. +When a commit tag in the file is missing in the destination or points at a different commit, `langstar prompt import` SHALL list the tag as a difference and exit with status 3. With `--update` it SHALL create or move the tag to the destination commit that matches the file, and add no commit when the latest commit already matches. #### Scenario: Destination tag points at an older commit @@ -148,7 +148,7 @@ When the destination record holds a commit for the destination workspace and the - **WHEN** a user edited the destination prompt in the web interface after the last import - **AND** runs `langstar prompt import my-prompt.json --workspace-id ` -- **THEN** langstar reports that the destination changed since the last recorded import and exits with status 2 +- **THEN** langstar reports that the destination changed since the last recorded import and exits with status 3 ### Requirement: Dry run an import @@ -162,12 +162,12 @@ When the destination record holds a commit for the destination workspace and the ### Requirement: Read the prompt back after import -After writing, `langstar prompt import` SHALL read the destination commit back with the model included and compare it with the file. When they differ it SHALL list the differences and exit with status 2. +After writing, `langstar prompt import` SHALL read the destination commit back with the model included and compare it with the file. When they differ it SHALL list the differences and exit with status 3. #### Scenario: LangSmith drops a field - **WHEN** the destination commit read back after import lacks a field the file holds -- **THEN** langstar lists that field as a difference and exits with status 2 +- **THEN** langstar lists that field as a difference and exits with status 3 ### Requirement: Compare a prompt with its export file @@ -181,22 +181,22 @@ After writing, `langstar prompt import` SHALL read the destination commit back w #### Scenario: Model temperature changed - **WHEN** someone changed the model temperature in the workspace's prompt -- **THEN** langstar lists the temperature's JSON path with both values and exits with status 2 +- **THEN** langstar lists the temperature's JSON path with both values and exits with status 3 #### Scenario: Repo readme changed - **WHEN** someone changed the readme of the workspace's prompt -- **THEN** langstar lists the readme's JSON path with both values and exits with status 2 +- **THEN** langstar lists the readme's JSON path with both values and exits with status 3 ### Requirement: Verify that a prompt can run in a workspace -`langstar prompt verify ` SHALL do what `compare` does and also check that the workspace has every secret in `requires.secrets`. It SHALL name each missing secret and exit with status 2 when any secret is missing. It SHALL also look for a saved model configuration whose settings `id` ends in the class name `requires.model_provider` holds, and report a missing one as a note that leaves the exit status unchanged, because the prompt runs with the model its manifest carries. +`langstar prompt verify ` SHALL do what `compare` does and also check that the workspace has every secret in `requires.secrets`. It SHALL name each missing secret and exit with status 3 when any secret is missing. It SHALL also look for a saved model configuration whose settings `id` ends in the class name `requires.model_provider` holds, and report a missing one as a note that leaves the exit status unchanged, because the prompt runs with the model its manifest carries. #### Scenario: Secret missing in the destination - **WHEN** the destination workspace has no secret named `OPENAI_API_KEY` - **AND** the file lists `OPENAI_API_KEY` under `requires.secrets` -- **THEN** langstar reports `OPENAI_API_KEY` as missing and exits with status 2 +- **THEN** langstar reports `OPENAI_API_KEY` as missing and exits with status 3 #### Scenario: No saved model configuration for the provider @@ -205,12 +205,17 @@ After writing, `langstar prompt import` SHALL read the destination commit back w - **AND** every secret in `requires.secrets` exists and the prompt matches the file - **THEN** langstar notes that no saved model configuration matches `ChatOpenAI` and exits with status 0 -### Requirement: Exit with 0 for a match, 2 for a difference and 1 for an error +### Requirement: Exit with 0 for a match, 3 for a difference and 1 for an error -`prompt compare`, `prompt verify` and `prompt import` SHALL exit with status 0 when the prompt matches and every check passes, 2 when they find a difference or a failed check, and 1 on any other error. With `-f json` each SHALL print one JSON object with `status`, `differences` and `checks`. +`prompt compare`, `prompt verify` and `prompt import` SHALL exit with status 0 when the prompt matches and every check passes, 3 when they find a difference or a failed check, and 1 on any other error. A usage error that clap rejects, such as a mistyped flag, SHALL keep clap's status 2. With `-f json` each SHALL print one JSON object with `status`, `differences` and `checks`. #### Scenario: Machine-readable compare - **WHEN** a user runs `langstar -f json prompt compare my-prompt.json` - **THEN** stdout holds one JSON object with `status`, `differences` and `checks` - **AND** progress messages go to stderr + +#### Scenario: Mistyped flag + +- **WHEN** a user runs `langstar prompt compare my-prompt.json --wrkspace-id ` +- **THEN** langstar prints clap's usage error and exits with status 2 diff --git a/openspec/changes/export-import-prompts-and-assistants/tasks.md b/openspec/changes/export-import-prompts-and-assistants/tasks.md index 5c84f54d6..cfb29d0d2 100644 --- a/openspec/changes/export-import-prompts-and-assistants/tasks.md +++ b/openspec/changes/export-import-prompts-and-assistants/tasks.md @@ -4,7 +4,7 @@ Each numbered group is one pull request, in the order listed. ## 1. Check the API round trip before building on it -- [ ] 1.1 Extract the commit get, commit create, repo update (`PATCH /api/v1/repos/{owner}/{repo}`), repo tags (`/api/v1/repos/{owner}/{repo}/tags` and `/api/v1/repos/{owner}/{repo}/tags/{tag_name}`), workspace secrets and playground settings endpoints from `reference/openapi/langchain/langsmith/openapi.json` into `reference/api-specs/langsmith/` and add their jq queries to `FRAGMENTS.md`; verify by rerunning each query and getting the same file +- [ ] 1.1 Extract the commit get, commit list (`GET /api/v1/commits/{owner}/{repo}`), commit create, repo update (`PATCH /api/v1/repos/{owner}/{repo}`), repo tags (`/api/v1/repos/{owner}/{repo}/tags` and `/api/v1/repos/{owner}/{repo}/tags/{tag_name}`), workspace secrets and playground settings endpoints from `reference/openapi/langchain/langsmith/openapi.json` into `reference/api-specs/langsmith/` and add their jq queries to `FRAGMENTS.md`; verify by rerunning each query and getting the same file - [ ] 1.2 Write `reference/research/807-openapi-validation.md`, recording from the live API: the manifest returned with `include=model` for a structured prompt with a model, the outer `id` of that manifest and the `model_provider` value returned beside it, whether pushing that manifest to a second workspace returns the same model, what pushing an identical manifest returns, what the Agent Server returns for `context`, whether an Agent Server assistant update that sends a metadata key with a null value removes that key, whether an update that sends `config` or `context` replaces the assistant's object or merges into it, whether an update can clear an assistant's description, and what `POST /api/v1/prompts/invoke_prompt` returns for a prompt with a model; verify that every claim in the report names the request that produced it ## 2. Test environments @@ -17,49 +17,49 @@ Each numbered group is one pull request, in the order listed. - [ ] 3.1 Add a commit read that sends `include=model` and returns `description`, `model_config` and `model_provider` to `sdk/src/prompts.rs`; verify with an httpmock test that asserts the query string and the parsed fields, and an integration test that reads a prompt with a model - [ ] 3.2 Add the commit's `id` to `CommitData` in `sdk/src/prompts.rs`, a call that lists a prompt's commits with their `id` and hash (`GET /api/v1/commits/{owner}/{repo}`), and calls that list, create and move commit tags, which name their commit by that `id` in `POST /api/v1/repos/{owner}/{repo}/tags` and `PATCH /api/v1/repos/{owner}/{repo}/tags/{tag_name}`; verify with httpmock tests that assert `commit_id` in each request body, and an integration test that pushes two commits, tags the first, finds the second commit's `id` through the commit list, moves the tag to it, and reads the tag back through the SDK -- [ ] 3.3 Add `context` and `version` to `Assistant`, `context` and `graph_id` to `UpdateAssistantRequest`, `context` to `CreateAssistantRequest`, `assistant_id` and `if_exists` to `CreateAssistantRequest`, and a `graph_id` filter to `AssistantSearchRequest`, in `sdk/src/assistants.rs`; verify with httpmock tests of the request bodies and an integration test that creates an assistant with context, reads it back with its version, finds it by graph ID, and updates its graph ID and reads the new graph ID back +- [ ] 3.3 Add `context` and `version` to `Assistant`, `context` and `graph_id` to `UpdateAssistantRequest`, `context` to `CreateAssistantRequest`, and a `graph_id` filter to `AssistantSearchRequest`, in `sdk/src/assistants.rs`; verify with httpmock tests of the request bodies and an integration test that creates an assistant with context, reads it back with its version, finds it by graph ID, and updates its graph ID and reads the new graph ID back - [ ] 3.4 Add `description` to `CommitRequest`, and a repo update that sends `PATCH /api/v1/repos/{owner}/{repo}` with `description`, `readme`, `tags` and `is_public`, to `sdk/src/prompts.rs`; verify with httpmock tests of the request bodies and an integration test that pushes a commit with a description, updates the repo's description, readme, tags and visibility, and reads each back ## 4. Shared code for the prompt and assistant commands: the export file, the comparison and the report - [ ] 4.1 Add the export file types, with sorted keys and no export time, and the destination record file, to a new module in `cli/src/`; verify with unit tests that build equivalent nested values with keys inserted in different orders, serialize each, and assert identical text with keys sorted at every level, and that updating one destination leaves the others unchanged - [ ] 4.2 Add the JSON comparison that drops server-set fields and reports differences by JSON Pointer; verify with unit tests on nested objects, arrays, missing keys and ignored fields -- [ ] 4.3 Add the shared result report and the exit statuses 0, 2 and 1, printing JSON to stdout and progress to stderr; verify with unit tests of the report and a CLI test that checks the exit status +- [ ] 4.3 Add the shared result report and the exit statuses 0, 3 and 1, printing JSON to stdout and progress to stderr, and leave clap's status 2 for usage errors; verify with unit tests of the report and CLI tests that a difference exits with status 3 and that an unknown flag still exits with status 2 - [ ] 4.4 Add the secret reference walk that lists secret names from a manifest; verify with unit tests on a copy of the `prompt_with_model` manifest from the LangSmith Python SDK's integration tests, which `references/langsmith-sdk.md` links, and on a manifest with no secret reference ## 5. prompt export -- [ ] 5.1 Add `langstar prompt export` with `--commit`, `--output` and the scoping flags; verify with integration tests that create prompts through the SDK (a `json_schema` prompt, a `function_calling` prompt with a nested schema, a prompt with several messages and a placeholder, and a prompt with a model and metadata), export each through the CLI, and assert every field of the file against the SDK's read of the commit, and an export with `--commit prod` of a prompt whose `prod` tag points at an earlier commit, asserting that `source.commit_hash` and the manifest are that earlier commit's +- [ ] 5.1 Add `langstar prompt export` with `--commit`, `--output` and the scoping flags; verify with integration tests that create prompts through the SDK (a `json_schema` prompt, a `function_calling` prompt with a nested schema, a prompt with several messages and a placeholder, and a prompt with a model and metadata), export each through the CLI, and assert every field of the file against the SDK's read of the commit, an export with `--commit prod` of a prompt whose `prod` tag points at an earlier commit, asserting that `source.commit_hash` and the manifest are that earlier commit's, and an error test that exporting a handle no prompt in the workspace has exits with status 1, names the handle and writes no file - [ ] 5.2 Document `prompt export` in `docs/usage/` and the README; verify that each documented command runs as written ## 6. prompt compare and verify -- [ ] 6.1 Add `langstar prompt compare`; verify with integration tests that export a prompt, change its model temperature through the SDK, and assert the reported path, both values and exit status 2, and that a changed commit description and a changed repo readme are each reported the same way +- [ ] 6.1 Add `langstar prompt compare`; verify with integration tests that export a prompt, change its model temperature through the SDK, and assert the reported path, both values and exit status 3, and that a changed commit description and a changed repo readme are each reported the same way - [ ] 6.2 Add `langstar prompt verify` with the secret check and the saved model configuration note, matching `requires.model_provider` against the class name at the end of each configuration's settings `id`; verify with integration tests against a workspace that lacks a listed secret and one that has it, and a test that a missing saved model configuration is reported and leaves the exit status at 0 - [ ] 6.3 Document compare and verify; verify that each documented command runs as written ## 7. prompt import -- [ ] 7.1 Add `langstar prompt import` with required `--workspace-id`, running the checks of `prompt verify` from 6.2 before it writes, creating the repo, the commit and its tags, then reading the commit back; verify with integration tests that export from the test workspace, import into the destination workspace, and compare the full source and destination commits through the SDK with the model included, the commit description among them, and the repo's description, readme, tags and visibility, for every prompt shape from 5.1, that a file whose `requires.secrets` names a secret the destination workspace lacks creates no prompt, names the secret and exits with status 2, and a CLI test that runs import without `--workspace-id` while `LANGSMITH_WORKSPACE_ID` is set and asserts that it exits with status 1, names the missing flag and creates no prompt -- [ ] 7.2 Add `--update`, `--dry-run`, `--record` and the destination record; verify with integration tests that a repeated import creates no commit, that a changed destination is refused without `--update`, that an import after someone pushed a commit to the destination since the recorded import says the destination changed since the last import and exits with status 2, that `--update` adds one commit with the right parent and the file's commit description, that `--update` against a destination whose repo description, readme, tags or visibility differ while the manifest matches updates the repo and adds no commit, that `--update` against a destination whose `prod` tag points at an earlier commit while the manifest matches moves the tag and adds no commit, that `--dry-run` lists the move, and that `--dry-run` writes nothing +- [ ] 7.1 Add `langstar prompt import` with required `--workspace-id`, which it declares optional like the other `langstar prompt` subcommands and checks itself, running the checks of `prompt verify` from 6.2 before it writes, creating the repo, the commit and its tags, then reading the commit back; verify with integration tests that export from the test workspace, import into the destination workspace, and compare the full source and destination commits through the SDK with the model included, the commit description among them, and the repo's description, readme, tags and visibility, for every prompt shape from 5.1, that a file whose `requires.secrets` names a secret the destination workspace lacks creates no prompt, names the secret and exits with status 3, and a CLI test that runs import without `--workspace-id` while `LANGSMITH_WORKSPACE_ID` is set and asserts that it exits with status 1, names the missing flag and creates no prompt +- [ ] 7.2 Add `--update`, `--dry-run`, `--record` and the destination record; verify with integration tests that a repeated import creates no commit, that a changed destination is refused without `--update`, that an import after someone pushed a commit to the destination since the recorded import says the destination changed since the last import and exits with status 3, that `--update` adds one commit with the right parent and the file's commit description, that `--update` against a destination whose repo description, readme, tags or visibility differ while the manifest matches updates the repo and adds no commit, that `--update` against a destination whose `prod` tag points at an earlier commit while the manifest matches moves the tag and adds no commit, that `--dry-run` lists the move, and that `--dry-run` writes nothing - [ ] 7.3 Add the structured output check as a mise task that pulls the imported prompt with `include_model=True`, invokes it and validates the result against the schema; verify by running it from the import integration test for both methods - [ ] 7.4 Document prompt import, the workspace scope of prompts and setting up destination credentials; verify that each documented command runs as written ## 8. assistant export -- [ ] 8.1 Add `langstar assistant export` with `--output` and `--prompt-ref`; verify with integration tests that create an assistant with configuration, context and metadata through the SDK and assert every field of the exported file, and an error test for a pointer that names nothing +- [ ] 8.1 Add `langstar assistant export` with `--output` and `--prompt-ref`; verify with integration tests that create an assistant with configuration, context and metadata through the SDK and assert every field of the exported file, an error test for a pointer that names nothing, and an error test that exporting an assistant ID the deployment lacks exits with status 1, names the ID and writes no file - [ ] 8.2 Document assistant export; verify that each documented command runs as written ## 9. assistant compare and verify -- [ ] 9.1 Add `langstar assistant compare` with `--assistant-id`, and the lookup of the recorded assistant in the destination record from 4.1; verify with an integration test that exports an assistant, changes its metadata through the SDK, runs compare with `--assistant-id` set to that assistant, and asserts the reported path and exit status 2, and a unit test that compare reads the assistant ID from a destination record written by the 4.1 types +- [ ] 9.1 Add `langstar assistant compare` with `--assistant-id`, and the lookup of the recorded assistant in the destination record from 4.1; verify with an integration test that exports an assistant, changes its metadata through the SDK, runs compare with `--assistant-id` set to that assistant, and asserts the reported path and exit status 3, and a unit test that compare reads the assistant ID from a destination record written by the 4.1 types - [ ] 9.2 Add `langstar assistant verify` with `--assistant-id`, `--workspace-id` for the workspace that holds the prompts, the graph check, and the prompt check, which reads `GET /api/v1/commits/{owner}/{repo}/{tag}`; verify with integration tests that run verify with `--assistant-id` against an exported assistant, one with a file whose graph ID the deployment lacks, one with a missing prompt tag, and one that passes `--workspace-id` set to the destination workspace - [ ] 9.3 Document assistant compare and verify; verify that each documented command runs as written ## 10. assistant import -- [ ] 10.1 Add `langstar assistant import` with `--workspace-id`, the graph check and prompt check of `assistant verify` from 9.2 run before it writes, creation, read-back, the destination record and `--record`; verify with an integration test that creates a tagged prompt in the test workspace, imports an assistant that refers to it into the destination deployment and checks name, graph ID, configuration, context, metadata and prompt references through the Agent Server API, that `--record ` writes the destination record to that path, and that a file whose graph ID the deployment lacks and a file with a missing prompt tag each create no assistant and exit with status 2 -- [ ] 10.2 Add `--update`, `--assistant-id` and `--dry-run`; verify with integration tests that a repeated import creates no assistant, that `--update` changes the recorded one, its graph ID among the fields, that `--update --assistant-id ` naming an assistant other than the recorded one updates that assistant, leaves the recorded one unchanged and writes the new ID to the destination record, that `--update` of an assistant with metadata `{"kept": 1, "stale": 2}` from a file with `{"kept": 1}` leaves `{"kept": 1}` or reports `/metadata/stale` and exits with status 2, whichever task 1.2 found the server does, that `--update` of an assistant with a description and context from a file with neither clears both or reports `/description` and `/context` and exits with status 2, whichever task 1.2 found the server does, that a deleted recorded assistant exits with status 1, and that `--dry-run` creates no assistant, changes none and leaves the destination record as it was +- [ ] 10.1 Add `langstar assistant import` with `--workspace-id`, the graph check and prompt check of `assistant verify` from 9.2 run before it writes, creation, read-back, the destination record and `--record`; verify with an integration test that creates a tagged prompt in the test workspace, imports an assistant that refers to it into the destination deployment and checks name, graph ID, configuration, context, metadata and prompt references through the Agent Server API, that `--record ` writes the destination record to that path, and that a file whose graph ID the deployment lacks and a file with a missing prompt tag each create no assistant and exit with status 3 +- [ ] 10.2 Add `--update`, `--assistant-id` and `--dry-run`; verify with integration tests that a repeated import creates no assistant, that `--update` changes the recorded one, its graph ID among the fields, that `--update --assistant-id ` naming an assistant other than the recorded one updates that assistant, leaves the recorded one unchanged and writes the new ID to the destination record, that `--update` of an assistant with metadata `{"kept": 1, "stale": 2}` from a file with `{"kept": 1}` leaves `{"kept": 1}` or reports `/metadata/stale` and exits with status 3, whichever task 1.2 found the server does, that `--update` of an assistant with a description and context from a file with neither clears both or reports `/description` and `/context` and exits with status 3, whichever task 1.2 found the server does, that a deleted recorded assistant exits with status 1, and that `--dry-run` creates no assistant, changes none and leaves the destination record as it was - [ ] 10.3 Add an end-to-end test that imports prompts into the destination workspace and then an assistant that refers to them onto the destination deployment, and runs `assistant verify`; verify that it passes and that it fails after deleting one prompt tag - [ ] 10.4 Document assistant import, the deployment scope of assistants, and how export and import relate to the `create`, `get` and `update` commands of [#728 Implement CRUD command structure (create, get, update) for prompt commands](https://github.com/codekiln/langstar/issues/728); verify that each documented command runs as written From 2a70a295f22d0957805791d953eb11d75607dc5c Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Tue, 13 Oct 2026 19:26:19 -0400 Subject: [PATCH 10/90] =?UTF-8?q?=F0=9F=93=9A=20docs(openspec):=20point=20?= =?UTF-8?q?the=20end-to-end=20test=20at=20the=20destination=20workspace,?= =?UTF-8?q?=20test=20requires,=20and=20let=20group=202=20go=20ahead=20with?= =?UTF-8?q?out=20task=202.1?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit #807 Export, import, compare, and verify prompts and assistants across LangSmith environments Co-Authored-By: Claude Opus 5.5 --- .../export-import-prompts-and-assistants/design.md | 2 +- .../changes/export-import-prompts-and-assistants/tasks.md | 8 ++++---- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/openspec/changes/export-import-prompts-and-assistants/design.md b/openspec/changes/export-import-prompts-and-assistants/design.md index 40ea4b005..b4c4d56af 100644 --- a/openspec/changes/export-import-prompts-and-assistants/design.md +++ b/openspec/changes/export-import-prompts-and-assistants/design.md @@ -63,7 +63,7 @@ Import writes the destination's IDs to `.destinations.json`, keyed by A prompt that exists in the destination gets a new commit only with `--update`, and its parent is the destination's latest commit, so LangSmith keeps the earlier history. Import sends the file's commit description with each commit it creates. With `--update`, import also sets the destination repo's description, readme, tags and visibility to the file's values through `PATCH /api/v1/repos/{owner}/{repo}` when they differ. When only those repo fields differ and the manifest matches, import updates the repo and adds no commit. A commit tag in the file that the destination lacks, or that points at another commit there, also counts as a difference; with `--update`, import creates or moves the tag to the destination commit that matches the file, and adds no commit when the latest commit already matches. The tag endpoints take a commit's `id`, which differs from its hash. The commit read and the repo read return no `id`, so import looks the commit up in the commit list, `GET /api/v1/commits/{owner}/{repo}`, whose entries carry both the `id` and the hash. -The Agent Server keeps the name, graph ID and description an update leaves out, and merges the metadata an update sends into the metadata the assistant already has, as `AssistantPatch` in `reference/openapi/langchain/agent-server/openapi.json` documents. So with `--update`, import asks for each removal and then checks that it happened. It sends the file's name, graph ID, configuration and context in full, an empty object for a configuration or context the file lacks, and a null value for each metadata key the destination has and the file lacks. `AssistantPatch` documents no way to clear a description, so [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8) tries a null and an empty description against the live server, along with a null metadata value and a replaced configuration and context. Import then reads the assistant back, and any key or field the file lacks that is still there shows up as a difference, with exit status 3. The Agent Server saves each update as a new version of the assistant. +With `--update`, import makes an existing assistant match the file and removes what the file lacks. The Agent Server keeps the name, graph ID and description an update leaves out, and merges the metadata an update sends into the metadata the assistant already has, as `AssistantPatch` in `reference/openapi/langchain/agent-server/openapi.json` documents. So import sends the file's name, graph ID, configuration and context in full, an empty object for a configuration or context the file lacks, and a null value for each metadata key the destination has and the file lacks. `AssistantPatch` documents no way to clear a description, so [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8) tries a null and an empty description against the live server, along with a null metadata value and a replaced configuration and context. Import then reads the assistant back, and any key or field the file lacks that is still there shows up as a difference, with exit status 3. The Agent Server saves each update as a new version of the assistant. Without `--update`, import lists the differences and exits with status 3. `--dry-run` reports the action import would take and writes nothing. diff --git a/openspec/changes/export-import-prompts-and-assistants/tasks.md b/openspec/changes/export-import-prompts-and-assistants/tasks.md index cfb29d0d2..ee816aabf 100644 --- a/openspec/changes/export-import-prompts-and-assistants/tasks.md +++ b/openspec/changes/export-import-prompts-and-assistants/tasks.md @@ -1,6 +1,6 @@ # Tasks -Each numbered group is one pull request, in the order listed. +Each numbered group is one pull request, in the order listed, except task 2.1. Task 2.1 waits on codekiln, so it goes in a pull request of its own that lands before group 6, and the rest of group 2 goes ahead without it. ## 1. Check the API round trip before building on it @@ -29,7 +29,7 @@ Each numbered group is one pull request, in the order listed. ## 5. prompt export -- [ ] 5.1 Add `langstar prompt export` with `--commit`, `--output` and the scoping flags; verify with integration tests that create prompts through the SDK (a `json_schema` prompt, a `function_calling` prompt with a nested schema, a prompt with several messages and a placeholder, and a prompt with a model and metadata), export each through the CLI, and assert every field of the file against the SDK's read of the commit, an export with `--commit prod` of a prompt whose `prod` tag points at an earlier commit, asserting that `source.commit_hash` and the manifest are that earlier commit's, and an error test that exporting a handle no prompt in the workspace has exits with status 1, names the handle and writes no file +- [ ] 5.1 Add `langstar prompt export` with `--commit`, `--output` and the scoping flags; verify with integration tests that create prompts through the SDK (a `json_schema` prompt, a `function_calling` prompt with a nested schema, a prompt with several messages and a placeholder, and a prompt with a model and metadata), export each through the CLI, and assert every field of the file against the SDK's read of the commit, an export with `--commit prod` of a prompt whose `prod` tag points at an earlier commit, asserting that `source.commit_hash` and the manifest are that earlier commit's, and an error test that exporting a handle no prompt in the workspace has exits with status 1, names the handle and writes no file, and that the file for the prompt with a model lists `OPENAI_API_KEY` under `requires.secrets` and `ChatOpenAI` under `requires.model_provider` and holds no secret value - [ ] 5.2 Document `prompt export` in `docs/usage/` and the README; verify that each documented command runs as written ## 6. prompt compare and verify @@ -40,7 +40,7 @@ Each numbered group is one pull request, in the order listed. ## 7. prompt import -- [ ] 7.1 Add `langstar prompt import` with required `--workspace-id`, which it declares optional like the other `langstar prompt` subcommands and checks itself, running the checks of `prompt verify` from 6.2 before it writes, creating the repo, the commit and its tags, then reading the commit back; verify with integration tests that export from the test workspace, import into the destination workspace, and compare the full source and destination commits through the SDK with the model included, the commit description among them, and the repo's description, readme, tags and visibility, for every prompt shape from 5.1, that a file whose `requires.secrets` names a secret the destination workspace lacks creates no prompt, names the secret and exits with status 3, and a CLI test that runs import without `--workspace-id` while `LANGSMITH_WORKSPACE_ID` is set and asserts that it exits with status 1, names the missing flag and creates no prompt +- [ ] 7.1 Add `langstar prompt import`, which exits with status 1 and names `--workspace-id` when the flag is missing, runs the checks of `prompt verify` from 6.2 before it writes, creates the repo, the commit and its tags, then reads the commit back; verify with integration tests that export from the test workspace, import into the destination workspace, and compare the full source and destination commits through the SDK with the model included, the commit description among them, and the repo's description, readme, tags and visibility, for every prompt shape from 5.1, that a file whose `requires.secrets` names a secret the destination workspace lacks creates no prompt, names the secret and exits with status 3, and a CLI test that runs import without `--workspace-id` while `LANGSMITH_WORKSPACE_ID` is set and asserts that it exits with status 1, names the missing flag and creates no prompt - [ ] 7.2 Add `--update`, `--dry-run`, `--record` and the destination record; verify with integration tests that a repeated import creates no commit, that a changed destination is refused without `--update`, that an import after someone pushed a commit to the destination since the recorded import says the destination changed since the last import and exits with status 3, that `--update` adds one commit with the right parent and the file's commit description, that `--update` against a destination whose repo description, readme, tags or visibility differ while the manifest matches updates the repo and adds no commit, that `--update` against a destination whose `prod` tag points at an earlier commit while the manifest matches moves the tag and adds no commit, that `--dry-run` lists the move, and that `--dry-run` writes nothing - [ ] 7.3 Add the structured output check as a mise task that pulls the imported prompt with `include_model=True`, invokes it and validates the result against the schema; verify by running it from the import integration test for both methods - [ ] 7.4 Document prompt import, the workspace scope of prompts and setting up destination credentials; verify that each documented command runs as written @@ -60,7 +60,7 @@ Each numbered group is one pull request, in the order listed. - [ ] 10.1 Add `langstar assistant import` with `--workspace-id`, the graph check and prompt check of `assistant verify` from 9.2 run before it writes, creation, read-back, the destination record and `--record`; verify with an integration test that creates a tagged prompt in the test workspace, imports an assistant that refers to it into the destination deployment and checks name, graph ID, configuration, context, metadata and prompt references through the Agent Server API, that `--record ` writes the destination record to that path, and that a file whose graph ID the deployment lacks and a file with a missing prompt tag each create no assistant and exit with status 3 - [ ] 10.2 Add `--update`, `--assistant-id` and `--dry-run`; verify with integration tests that a repeated import creates no assistant, that `--update` changes the recorded one, its graph ID among the fields, that `--update --assistant-id ` naming an assistant other than the recorded one updates that assistant, leaves the recorded one unchanged and writes the new ID to the destination record, that `--update` of an assistant with metadata `{"kept": 1, "stale": 2}` from a file with `{"kept": 1}` leaves `{"kept": 1}` or reports `/metadata/stale` and exits with status 3, whichever task 1.2 found the server does, that `--update` of an assistant with a description and context from a file with neither clears both or reports `/description` and `/context` and exits with status 3, whichever task 1.2 found the server does, that a deleted recorded assistant exits with status 1, and that `--dry-run` creates no assistant, changes none and leaves the destination record as it was -- [ ] 10.3 Add an end-to-end test that imports prompts into the destination workspace and then an assistant that refers to them onto the destination deployment, and runs `assistant verify`; verify that it passes and that it fails after deleting one prompt tag +- [ ] 10.3 Add an end-to-end test that imports prompts into the destination workspace, then imports an assistant that refers to them onto the destination deployment and runs `assistant verify`, passing `--workspace-id` set to the destination workspace to both assistant commands; verify that it passes, and that verify fails after deleting one prompt tag in the destination workspace - [ ] 10.4 Document assistant import, the deployment scope of assistants, and how export and import relate to the `create`, `get` and `update` commands of [#728 Implement CRUD command structure (create, get, update) for prompt commands](https://github.com/codekiln/langstar/issues/728); verify that each documented command runs as written ## Workflow follow-up From fa5f0b35bbde1b52cd4d80b780b14638616ef5e6 Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Tue, 13 Oct 2026 19:27:22 -0400 Subject: [PATCH 11/90] =?UTF-8?q?=F0=9F=93=9A=20docs(openspec):=20run=20on?= =?UTF-8?q?ly=20verify's=20dependency=20checks=20before=20import=20and=20l?= =?UTF-8?q?et=20update=20fall=20back=20to=20a=20reported=20difference?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit #807 Export, import, compare, and verify prompts and assistants across LangSmith environments Co-Authored-By: Claude Opus 5.5 --- .../changes/export-import-prompts-and-assistants/design.md | 2 +- .../changes/export-import-prompts-and-assistants/proposal.md | 2 +- .../specs/assistant/spec.md | 3 ++- .../export-import-prompts-and-assistants/specs/prompt/spec.md | 2 +- openspec/changes/export-import-prompts-and-assistants/tasks.md | 2 +- 5 files changed, 6 insertions(+), 5 deletions(-) diff --git a/openspec/changes/export-import-prompts-and-assistants/design.md b/openspec/changes/export-import-prompts-and-assistants/design.md index b4c4d56af..0709adeb1 100644 --- a/openspec/changes/export-import-prompts-and-assistants/design.md +++ b/openspec/changes/export-import-prompts-and-assistants/design.md @@ -79,7 +79,7 @@ Assistant verify checks the graph with `POST /assistants/search` filtered to the ### Import runs verify's checks before it writes -Before it writes anything, import runs the checks verify runs against the destination, because [#807 Export, import, compare, and verify prompts and assistants across LangSmith environments](https://github.com/codekiln/langstar/issues/807) asks import to check the destination's dependencies. Prompt import stops when the destination workspace lacks a secret in `requires.secrets`, and assistant import stops when the deployment does not serve the file's graph or the workspace, from `--workspace-id` or the configured workspace, lacks a prompt in `requires.prompts`. Each names what is missing, writes nothing and exits with status 3, and `--dry-run` reports the same failed checks. Import therefore writes a copy only when the destination has the secrets, graph and prompts that copy needs to run. In return, the user works in order: create the destination's secrets before importing a prompt, and import the prompts before importing an assistant that refers to them. Open Question 8 asks whether import should write anyway and report the failed checks afterwards. +Before it writes anything, import runs the dependency checks verify runs against the destination, because [#807 Export, import, compare, and verify prompts and assistants across LangSmith environments](https://github.com/codekiln/langstar/issues/807) asks import to check the destination's dependencies. It skips verify's comparison, since the destination may have no copy yet. Prompt import stops when the destination workspace lacks a secret in `requires.secrets`, and assistant import stops when the deployment does not serve the file's graph or the workspace, from `--workspace-id` or the configured workspace, lacks a prompt in `requires.prompts`. Each names what is missing, writes nothing and exits with status 3, and `--dry-run` reports the same failed checks. Import therefore writes a copy only when the destination has the secrets, graph and prompts that copy needs to run. In return, the user works in order: create the destination's secrets before importing a prompt, and import the prompts before importing an assistant that refers to them. Open Question 8 asks whether import should write anyway and report the failed checks afterwards. ### The user names an assistant's prompt references diff --git a/openspec/changes/export-import-prompts-and-assistants/proposal.md b/openspec/changes/export-import-prompts-and-assistants/proposal.md index db8a7a789..950b82a52 100644 --- a/openspec/changes/export-import-prompts-and-assistants/proposal.md +++ b/openspec/changes/export-import-prompts-and-assistants/proposal.md @@ -15,7 +15,7 @@ A prompt belongs to a workspace and an assistant belongs to a deployment, but bo - `langstar prompt compare ` reports the differences between the file and a prompt in a workspace. `langstar prompt verify ` also checks that the workspace has the secrets the prompt's model reads, and notes when the workspace has no saved model configuration for the model's provider. - `langstar assistant export`, `import`, `compare` and `verify` do the same for assistants on a deployment. The file holds the name, description, graph ID, configuration, context, metadata and any prompt names the assistant refers to. `verify` checks that the destination deployment has the graph and that the destination workspace has the referenced prompts. - Import records each destination ID (the new commit hash, or the new assistant ID) in a separate file next to the export, so importing the same file again finds the copy it made earlier instead of creating another one. -- Import runs the checks of `verify` before it writes, and writes nothing when one fails. +- Import runs the dependency checks of `verify` before it writes, and writes nothing when one fails. - `compare`, `verify` and `import` share one exit status scheme, 0 for a match, 3 for a difference or a failed check and 1 for any other error, and print a JSON report with `-f json`. Status 2 stays clap's usage error. ## Capabilities diff --git a/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md b/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md index 53e674e19..988b4c366 100644 --- a/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md +++ b/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md @@ -111,12 +111,13 @@ When the destination record holds an assistant ID for the deployment, `langstar ### Requirement: Remove on update what the file lacks -`langstar assistant import --update` SHALL remove each metadata key the destination assistant has and the file lacks, and clear a description or context the file lacks. When such a key or field remains after the update, it SHALL list it as a difference and exit with status 3. +`langstar assistant import --update` SHALL ask the Agent Server to remove each metadata key the destination assistant has and the file lacks, and to clear a description or context the file lacks, then read the assistant back. When such a key or field remains after the update, it SHALL list it as a difference and exit with status 3. #### Scenario: Remove a metadata key the file lacks - **WHEN** the recorded assistant's metadata is `{"kept": 1, "stale": 2}` - **AND** the file's metadata is `{"kept": 1}` +- **AND** the Agent Server removes a metadata key that an update sends with a null value - **AND** the user runs `langstar assistant import support-bot.json --deployment my-agent-prod --update` - **THEN** the assistant's metadata read back from the deployment is `{"kept": 1}` - **AND** langstar exits with status 0 diff --git a/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md index dfb4b2781..d3f2786cd 100644 --- a/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md +++ b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md @@ -65,7 +65,7 @@ The export file SHALL list, under `requires.secrets`, the name of every secret t ### Requirement: Check the destination before importing a prompt -Before it writes, `langstar prompt import` SHALL run the checks of `prompt verify` against the destination workspace. When the workspace lacks a secret in `requires.secrets` it SHALL write nothing, name each missing secret and exit with status 3. A missing saved model configuration SHALL be a note, as in `prompt verify`. +Before it writes, `langstar prompt import` SHALL run the secret check and the saved model configuration check of `prompt verify` against the destination workspace, and not its comparison, because the destination may have no prompt yet. When the workspace lacks a secret in `requires.secrets` it SHALL write nothing, name each missing secret and exit with status 3. A missing saved model configuration SHALL be a note, as in `prompt verify`. #### Scenario: Secret missing at import diff --git a/openspec/changes/export-import-prompts-and-assistants/tasks.md b/openspec/changes/export-import-prompts-and-assistants/tasks.md index ee816aabf..0d32fc940 100644 --- a/openspec/changes/export-import-prompts-and-assistants/tasks.md +++ b/openspec/changes/export-import-prompts-and-assistants/tasks.md @@ -40,7 +40,7 @@ Each numbered group is one pull request, in the order listed, except task 2.1. T ## 7. prompt import -- [ ] 7.1 Add `langstar prompt import`, which exits with status 1 and names `--workspace-id` when the flag is missing, runs the checks of `prompt verify` from 6.2 before it writes, creates the repo, the commit and its tags, then reads the commit back; verify with integration tests that export from the test workspace, import into the destination workspace, and compare the full source and destination commits through the SDK with the model included, the commit description among them, and the repo's description, readme, tags and visibility, for every prompt shape from 5.1, that a file whose `requires.secrets` names a secret the destination workspace lacks creates no prompt, names the secret and exits with status 3, and a CLI test that runs import without `--workspace-id` while `LANGSMITH_WORKSPACE_ID` is set and asserts that it exits with status 1, names the missing flag and creates no prompt +- [ ] 7.1 Add `langstar prompt import`, which exits with status 1 and names `--workspace-id` when the flag is missing, runs the secret and saved model configuration checks of `prompt verify` from 6.2 before it writes, creates the repo, the commit and its tags, then reads the commit back; verify with integration tests that export from the test workspace, import into the destination workspace, and compare the full source and destination commits through the SDK with the model included, the commit description among them, and the repo's description, readme, tags and visibility, for every prompt shape from 5.1, that a file whose `requires.secrets` names a secret the destination workspace lacks creates no prompt, names the secret and exits with status 3, and a CLI test that runs import without `--workspace-id` while `LANGSMITH_WORKSPACE_ID` is set and asserts that it exits with status 1, names the missing flag and creates no prompt - [ ] 7.2 Add `--update`, `--dry-run`, `--record` and the destination record; verify with integration tests that a repeated import creates no commit, that a changed destination is refused without `--update`, that an import after someone pushed a commit to the destination since the recorded import says the destination changed since the last import and exits with status 3, that `--update` adds one commit with the right parent and the file's commit description, that `--update` against a destination whose repo description, readme, tags or visibility differ while the manifest matches updates the repo and adds no commit, that `--update` against a destination whose `prod` tag points at an earlier commit while the manifest matches moves the tag and adds no commit, that `--dry-run` lists the move, and that `--dry-run` writes nothing - [ ] 7.3 Add the structured output check as a mise task that pulls the imported prompt with `include_model=True`, invokes it and validates the result against the schema; verify by running it from the import integration test for both methods - [ ] 7.4 Document prompt import, the workspace scope of prompts and setting up destination credentials; verify that each documented command runs as written From 6f59b000cbf70e0b6f8da3e831fb5340229f0126 Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Tue, 13 Oct 2026 19:30:17 -0400 Subject: [PATCH 12/90] =?UTF-8?q?=F0=9F=93=9A=20docs(openspec):=20say=20im?= =?UTF-8?q?port=20asks=20the=20Agent=20Server=20to=20remove=20what=20the?= =?UTF-8?q?=20file=20lacks?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit #807 Export, import, compare, and verify prompts and assistants across LangSmith environments Co-Authored-By: Claude Opus 5.5 --- openspec/changes/export-import-prompts-and-assistants/design.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/openspec/changes/export-import-prompts-and-assistants/design.md b/openspec/changes/export-import-prompts-and-assistants/design.md index 0709adeb1..86db1d6de 100644 --- a/openspec/changes/export-import-prompts-and-assistants/design.md +++ b/openspec/changes/export-import-prompts-and-assistants/design.md @@ -63,7 +63,7 @@ Import writes the destination's IDs to `.destinations.json`, keyed by A prompt that exists in the destination gets a new commit only with `--update`, and its parent is the destination's latest commit, so LangSmith keeps the earlier history. Import sends the file's commit description with each commit it creates. With `--update`, import also sets the destination repo's description, readme, tags and visibility to the file's values through `PATCH /api/v1/repos/{owner}/{repo}` when they differ. When only those repo fields differ and the manifest matches, import updates the repo and adds no commit. A commit tag in the file that the destination lacks, or that points at another commit there, also counts as a difference; with `--update`, import creates or moves the tag to the destination commit that matches the file, and adds no commit when the latest commit already matches. The tag endpoints take a commit's `id`, which differs from its hash. The commit read and the repo read return no `id`, so import looks the commit up in the commit list, `GET /api/v1/commits/{owner}/{repo}`, whose entries carry both the `id` and the hash. -With `--update`, import makes an existing assistant match the file and removes what the file lacks. The Agent Server keeps the name, graph ID and description an update leaves out, and merges the metadata an update sends into the metadata the assistant already has, as `AssistantPatch` in `reference/openapi/langchain/agent-server/openapi.json` documents. So import sends the file's name, graph ID, configuration and context in full, an empty object for a configuration or context the file lacks, and a null value for each metadata key the destination has and the file lacks. `AssistantPatch` documents no way to clear a description, so [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8) tries a null and an empty description against the live server, along with a null metadata value and a replaced configuration and context. Import then reads the assistant back, and any key or field the file lacks that is still there shows up as a difference, with exit status 3. The Agent Server saves each update as a new version of the assistant. +With `--update`, import makes an existing assistant match the file and asks the Agent Server to remove what the file lacks. The Agent Server keeps the name, graph ID and description an update leaves out, and merges the metadata an update sends into the metadata the assistant already has, as `AssistantPatch` in `reference/openapi/langchain/agent-server/openapi.json` documents. So import sends the file's name, graph ID, configuration and context in full, an empty object for a configuration or context the file lacks, and a null value for each metadata key the destination has and the file lacks. `AssistantPatch` documents no way to clear a description, so [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8) tries a null and an empty description against the live server, along with a null metadata value and a replaced configuration and context. Import then reads the assistant back, and any key or field the file lacks that is still there shows up as a difference, with exit status 3. The Agent Server saves each update as a new version of the assistant. Without `--update`, import lists the differences and exits with status 3. `--dry-run` reports the action import would take and writes nothing. From 337e0dceb62dd739e43688a20c63f2243cc797ad Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Tue, 13 Oct 2026 19:35:46 -0400 Subject: [PATCH 13/90] =?UTF-8?q?=F0=9F=93=9A=20docs(openspec):=20test=20t?= =?UTF-8?q?he=20JSON=20report=20in=20compare=20and=20verify,=20and=20name?= =?UTF-8?q?=20what=20import=20checks=20before=20it=20writes?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Task 4.3 tests only the report and its exit statuses, since group 4 adds no command that can find a difference. Tasks 6.1 and 9.2 now test the `-f json` report on stdout, progress on stderr and clap's status 2. design.md names the checks import runs, links open questions 7 and 8 by heading and links #807 by title. #807 Export, import, compare, and verify prompts and assistants across LangSmith environments Co-Authored-By: Claude Opus 5.5 --- .../design.md | 16 ++++++++-------- .../tasks.md | 6 +++--- 2 files changed, 11 insertions(+), 11 deletions(-) diff --git a/openspec/changes/export-import-prompts-and-assistants/design.md b/openspec/changes/export-import-prompts-and-assistants/design.md index 86db1d6de..181b0b18d 100644 --- a/openspec/changes/export-import-prompts-and-assistants/design.md +++ b/openspec/changes/export-import-prompts-and-assistants/design.md @@ -6,7 +6,7 @@ `langstar assistant` in `cli/src/commands/assistant.rs` takes `--deployment` on every subcommand and sends requests to that deployment's Agent Server. `AssistantClient` in `sdk/src/assistants.rs` has no `context` or `version` field and its search cannot filter by graph ID. -Langstar reads one API key and one workspace at a time, from flags, environment variables or the config file. A langstar error exits with status 1, and a usage error that clap rejects, such as a mistyped flag, exits with status 2. +Langstar reads one API key and one workspace at a time, from flags, environment variables or the config file. Langstar exits with status 1 when a command fails, and clap, the library that parses langstar's flags, exits with status 2 when it rejects a mistyped flag or a missing required one. ## Goals / Non-Goals @@ -57,7 +57,7 @@ Export reads the commit with `include=model`, the form the Go SDK uses (`referen ### Import records what it created in a separate file -Import writes the destination's IDs to `.destinations.json`, keyed by destination workspace ID for prompts and by deployment ID for assistants. The export file stays the same after any number of imports, and a fresh export never wipes out the record of earlier imports. A repeated import reads the record to find the copy it made. For a prompt, the handle already finds the copy, and the record tells langstar whether someone changed the destination since the last import. For an assistant, whose ID differs on every deployment, the record is the only link from the file to its copy. +Import writes the destination's IDs to `.destinations.json`, keyed by destination workspace ID for prompts and by deployment ID for assistants. The export file stays the same after any number of imports, and a fresh export never wipes out the record of earlier imports. A repeated import reads the record to find the copy it made. For a prompt, langstar finds the copy by its handle, and compares the commit hash in the record with the destination's latest commit to tell whether someone changed the prompt since the last import. For an assistant, whose ID differs on every deployment, the record is the only link from the file to its copy. ### Import changes an existing resource only with --update @@ -73,13 +73,13 @@ Compare reads the destination, removes the fields the server sets (IDs, version ### Verify checks what the copy needs to run -Prompt verify looks up each name in `requires.secrets` among the workspace's secrets (`GET /api/v1/workspaces/current/secrets`, which `langstar secrets list` already calls). It also looks for a saved model configuration (`GET /api/v1/playground-settings`, which `langstar model-config list` calls) whose settings `id` ends in the class name that `requires.model_provider` holds, such as `ChatOpenAI`. Export reads that class name from the last element of the chat model constructor's `id` in the manifest, such as `["langchain", "chat_models", "openai", "ChatOpenAI"]`. LangSmith also returns a `model_provider` field beside the manifest, and the file keeps it as `commit.model_provider`. No SDK or spec shows what that field holds, so [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8) records it. The prompt runs without a saved model configuration, because its manifest carries the model, so verify reports a missing one as a note and leaves the exit status alone; [#807 Export, import, compare, and verify prompts and assistants across LangSmith environments](https://github.com/codekiln/langstar/issues/807) asks for the report, and Open Question 7 asks whether to keep it. Verify prints the name of each missing secret and no secret value. Langstar finds secret names by walking the manifest for LangChain secret references, `{"lc": 1, "type": "secret", "id": ["OPENAI_API_KEY"]}`, the shape the Python SDK's `_manifest_has_secrets` matches. +Prompt verify looks up each name in `requires.secrets` among the workspace's secrets (`GET /api/v1/workspaces/current/secrets`, which `langstar secrets list` already calls). It also looks for a saved model configuration (`GET /api/v1/playground-settings`, which `langstar model-config list` calls) whose settings `id` ends in the class name that `requires.model_provider` holds, such as `ChatOpenAI`. Export reads that class name from the last element of the chat model constructor's `id` in the manifest, such as `["langchain", "chat_models", "openai", "ChatOpenAI"]`. LangSmith also returns a `model_provider` field beside the manifest, and the file keeps it as `commit.model_provider`. No SDK or spec shows what that field holds, so [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8) records it. The prompt runs without a saved model configuration, because its manifest carries the model, so verify reports a missing one as a note and leaves the exit status alone; [#807 Export, import, compare, and verify prompts and assistants across LangSmith environments](https://github.com/codekiln/langstar/issues/807) asks for the report, and [Should prompt verify report a missing saved model configuration?](#7---should-prompt-verify-report-a-missing-saved-model-configuration) asks whether to keep it. Verify prints the name of each missing secret and no secret value. Langstar finds secret names by walking the manifest for LangChain secret references, `{"lc": 1, "type": "secret", "id": ["OPENAI_API_KEY"]}`, the shape the Python SDK's `_manifest_has_secrets` matches. Assistant verify checks the graph with `POST /assistants/search` filtered to the graph ID; every graph on a deployment has a default assistant, so an empty result means the deployment does not serve the graph. It checks each prompt in `requires.prompts`, such as `support-system:prod`, by asking LangSmith for the commit that tag points at (`GET /api/v1/commits/{owner}/{repo}/{tag}`, the request `langstar prompt pull --commit prod` already sends). A 404 means the workspace lacks the prompt or its tag. -### Import runs verify's checks before it writes +### Import checks the destination's secrets, graph and prompts before it writes -Before it writes anything, import runs the dependency checks verify runs against the destination, because [#807 Export, import, compare, and verify prompts and assistants across LangSmith environments](https://github.com/codekiln/langstar/issues/807) asks import to check the destination's dependencies. It skips verify's comparison, since the destination may have no copy yet. Prompt import stops when the destination workspace lacks a secret in `requires.secrets`, and assistant import stops when the deployment does not serve the file's graph or the workspace, from `--workspace-id` or the configured workspace, lacks a prompt in `requires.prompts`. Each names what is missing, writes nothing and exits with status 3, and `--dry-run` reports the same failed checks. Import therefore writes a copy only when the destination has the secrets, graph and prompts that copy needs to run. In return, the user works in order: create the destination's secrets before importing a prompt, and import the prompts before importing an assistant that refers to them. Open Question 8 asks whether import should write anyway and report the failed checks afterwards. +Before it writes anything, import runs the dependency checks verify runs against the destination, because [#807 Export, import, compare, and verify prompts and assistants across LangSmith environments](https://github.com/codekiln/langstar/issues/807) asks import to check the destination's dependencies. It skips verify's comparison, since the destination may have no copy yet. Prompt import stops when the destination workspace lacks a secret in `requires.secrets`, and assistant import stops when the deployment does not serve the file's graph or the workspace, from `--workspace-id` or the configured workspace, lacks a prompt in `requires.prompts`. Each names what is missing, writes nothing and exits with status 3, and `--dry-run` reports the same failed checks. Import therefore writes a copy only when the destination has the secrets, graph and prompts that copy needs to run. A user therefore has to create the destination workspace's secrets before importing a prompt, and import the prompts before importing an assistant that refers to them. [Should a failed dependency check stop import before it writes?](#8---should-a-failed-dependency-check-stop-import-before-it-writes) asks whether import should write anyway and report the failed checks afterwards. ### The user names an assistant's prompt references @@ -148,20 +148,20 @@ None. ### 6 - Should prompt import refuse to run without --workspace-id even when LANGSMITH_WORKSPACE_ID is set? > Context from the drafting agent, for question 6. -> Issue #807 asks for an explicit destination workspace. Requiring the flag means a leftover environment variable can never send a prompt to the wrong workspace, at the cost of one more flag in scripts. Assistant import already requires `--deployment`. Recommendation: require the flag. +> [#807 Export, import, compare, and verify prompts and assistants across LangSmith environments](https://github.com/codekiln/langstar/issues/807) asks for an explicit destination workspace. Requiring the flag means a leftover environment variable can never send a prompt to the wrong workspace, at the cost of one more flag in scripts. Assistant import already requires `--deployment`. Recommendation: require the flag. ### 7 - Should prompt verify report a missing saved model configuration? > Context from the drafting agent, for question 7. -> Issue #807 asks import to report missing model configurations, so the draft has prompt verify look for a saved model configuration (`GET /api/v1/playground-settings`) whose settings `id` ends in the class name the export file holds in `requires.model_provider`, such as `ChatOpenAI`. Nothing fails at run time without one, because the prompt's manifest carries the model and its settings; the secret check is the one that finds a prompt that cannot authenticate. Reporting it as a failure would make verify exit with 3 for a prompt that runs. Recommendation: report a missing saved model configuration as a note that leaves the exit status alone. +> [#807 Export, import, compare, and verify prompts and assistants across LangSmith environments](https://github.com/codekiln/langstar/issues/807) asks import to report missing model configurations, so the draft has prompt verify look for a saved model configuration (`GET /api/v1/playground-settings`) whose settings `id` ends in the class name the export file holds in `requires.model_provider`, such as `ChatOpenAI`. Nothing fails at run time without one, because the prompt's manifest carries the model and its settings; the secret check is the one that finds a prompt that cannot authenticate. Reporting it as a failure would make verify exit with 3 for a prompt that runs. Recommendation: report a missing saved model configuration as a note that leaves the exit status alone. ### 8 - Should a failed dependency check stop import before it writes? > Context from the drafting agent, for question 8. -> Issue #807 asks import to check the destination's dependencies: the secrets a prompt's model reads, and an assistant's graph and referenced prompts. The draft runs those checks before import writes and writes nothing when one fails, so a user never has to find and delete a copy that cannot run. The cost is order: the destination workspace needs its secrets before a prompt import, and its prompts before an assistant import. The other choice writes the copy anyway and reports the failed checks with status 3 afterwards, which lets a user import in any order and fix the destination later. Recommendation: stop before writing. +> [#807 Export, import, compare, and verify prompts and assistants across LangSmith environments](https://github.com/codekiln/langstar/issues/807) asks import to check the destination's dependencies: the secrets a prompt's model reads, and an assistant's graph and referenced prompts. The draft runs those checks before import writes and writes nothing when one fails, so a user never has to find and delete a copy that cannot run. The cost is order: the destination workspace needs its secrets before a prompt import, and its prompts before an assistant import. The other choice writes the copy anyway and reports the failed checks with status 3 afterwards, which lets a user import in any order and fix the destination later. Recommendation: stop before writing. diff --git a/openspec/changes/export-import-prompts-and-assistants/tasks.md b/openspec/changes/export-import-prompts-and-assistants/tasks.md index 0d32fc940..0e31c7cbb 100644 --- a/openspec/changes/export-import-prompts-and-assistants/tasks.md +++ b/openspec/changes/export-import-prompts-and-assistants/tasks.md @@ -24,7 +24,7 @@ Each numbered group is one pull request, in the order listed, except task 2.1. T - [ ] 4.1 Add the export file types, with sorted keys and no export time, and the destination record file, to a new module in `cli/src/`; verify with unit tests that build equivalent nested values with keys inserted in different orders, serialize each, and assert identical text with keys sorted at every level, and that updating one destination leaves the others unchanged - [ ] 4.2 Add the JSON comparison that drops server-set fields and reports differences by JSON Pointer; verify with unit tests on nested objects, arrays, missing keys and ignored fields -- [ ] 4.3 Add the shared result report and the exit statuses 0, 3 and 1, printing JSON to stdout and progress to stderr, and leave clap's status 2 for usage errors; verify with unit tests of the report and CLI tests that a difference exits with status 3 and that an unknown flag still exits with status 2 +- [ ] 4.3 Add the shared result report and the exit statuses 0, 3 and 1, printing JSON to stdout and progress to stderr, and leave clap's status 2 for usage errors; verify with unit tests of the report and of the exit status each result gives - [ ] 4.4 Add the secret reference walk that lists secret names from a manifest; verify with unit tests on a copy of the `prompt_with_model` manifest from the LangSmith Python SDK's integration tests, which `references/langsmith-sdk.md` links, and on a manifest with no secret reference ## 5. prompt export @@ -34,7 +34,7 @@ Each numbered group is one pull request, in the order listed, except task 2.1. T ## 6. prompt compare and verify -- [ ] 6.1 Add `langstar prompt compare`; verify with integration tests that export a prompt, change its model temperature through the SDK, and assert the reported path, both values and exit status 3, and that a changed commit description and a changed repo readme are each reported the same way +- [ ] 6.1 Add `langstar prompt compare`; verify with integration tests that export a prompt, change its model temperature through the SDK, and assert the reported path, both values and exit status 3, and that a changed commit description and a changed repo readme are each reported the same way, and CLI tests that `langstar -f json prompt compare` prints one JSON object with `status`, `differences` and `checks` to stdout and its progress to stderr, and that `prompt compare` with a mistyped flag exits with status 2 - [ ] 6.2 Add `langstar prompt verify` with the secret check and the saved model configuration note, matching `requires.model_provider` against the class name at the end of each configuration's settings `id`; verify with integration tests against a workspace that lacks a listed secret and one that has it, and a test that a missing saved model configuration is reported and leaves the exit status at 0 - [ ] 6.3 Document compare and verify; verify that each documented command runs as written @@ -53,7 +53,7 @@ Each numbered group is one pull request, in the order listed, except task 2.1. T ## 9. assistant compare and verify - [ ] 9.1 Add `langstar assistant compare` with `--assistant-id`, and the lookup of the recorded assistant in the destination record from 4.1; verify with an integration test that exports an assistant, changes its metadata through the SDK, runs compare with `--assistant-id` set to that assistant, and asserts the reported path and exit status 3, and a unit test that compare reads the assistant ID from a destination record written by the 4.1 types -- [ ] 9.2 Add `langstar assistant verify` with `--assistant-id`, `--workspace-id` for the workspace that holds the prompts, the graph check, and the prompt check, which reads `GET /api/v1/commits/{owner}/{repo}/{tag}`; verify with integration tests that run verify with `--assistant-id` against an exported assistant, one with a file whose graph ID the deployment lacks, one with a missing prompt tag, and one that passes `--workspace-id` set to the destination workspace +- [ ] 9.2 Add `langstar assistant verify` with `--assistant-id`, `--workspace-id` for the workspace that holds the prompts, the graph check, and the prompt check, which reads `GET /api/v1/commits/{owner}/{repo}/{tag}`; verify with integration tests that run verify with `--assistant-id` against an exported assistant, one with a file whose graph ID the deployment lacks, one with a missing prompt tag, and one that passes `--workspace-id` set to the destination workspace, and a CLI test that `langstar -f json assistant verify` prints one JSON object with `status`, `differences` and `checks` to stdout - [ ] 9.3 Document assistant compare and verify; verify that each documented command runs as written ## 10. assistant import From 80b2502c57991cdc5461dd9af305272e3949b733 Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Tue, 13 Oct 2026 19:36:58 -0400 Subject: [PATCH 14/90] =?UTF-8?q?=F0=9F=93=9A=20docs(openspec):=20pass=20t?= =?UTF-8?q?he=20model=20provider=20key=20to=20pull=5Fprompt=20as=20a=20sec?= =?UTF-8?q?rets=20map?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The pinned LangSmith Python SDK's pull_prompt reads no secret from the environment by default and raises when a manifest holds a secret reference and no secrets map arrives, so the structured output script passes the key it reads from the environment as a secrets map. #807 Export, import, compare, and verify prompts and assistants across LangSmith environments Co-Authored-By: Claude Opus 5.5 --- openspec/changes/export-import-prompts-and-assistants/design.md | 2 +- .../references/langsmith-sdk.md | 2 +- openspec/changes/export-import-prompts-and-assistants/tasks.md | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/openspec/changes/export-import-prompts-and-assistants/design.md b/openspec/changes/export-import-prompts-and-assistants/design.md index 181b0b18d..31f7cde46 100644 --- a/openspec/changes/export-import-prompts-and-assistants/design.md +++ b/openspec/changes/export-import-prompts-and-assistants/design.md @@ -91,7 +91,7 @@ Status 0 means the resource matches and every check passed. Status 3 means langs ### A Python script checks that an imported structured prompt still returns structured data -A mise task runs a short Python script that pulls the imported prompt with `include_model=True`, invokes it, and checks that the result matches the prompt's schema. The script exists because the LangSmith API has no documented way to run a prompt with its model: the one prompt endpoint in the OpenAPI spec, `POST /api/v1/prompts/invoke_prompt`, takes message templates and inputs with no model or schema, and no LangSmith SDK calls it. [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8) tries that endpoint against the live API in case it does more than the spec says. The script needs a model provider API key, which codekiln has to create and add to fnox and CI before the test can run. +A mise task runs a short Python script that pulls the imported prompt with `include_model=True`, invokes it, and checks that the result matches the prompt's schema. The script passes `secrets={"OPENAI_API_KEY": }` to [`pull_prompt`](https://github.com/langchain-ai/langsmith-sdk/blob/47c553caf08c1989e2be3ff1886da3eb801f709f/python/langsmith/client.py#L10228), because the LangSmith Python SDK reads no secret from the environment unless asked and raises when a manifest holds a secret reference and no `secrets` map arrives. The script exists because the LangSmith API has no documented way to run a prompt with its model: the one prompt endpoint in the OpenAPI spec, `POST /api/v1/prompts/invoke_prompt`, takes message templates and inputs with no model or schema, and no LangSmith SDK calls it. [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8) tries that endpoint against the live API in case it does more than the spec says. The script needs a model provider API key, which codekiln has to create and add to fnox and CI before the test can run. ### Export and import use the same SDK calls as the planned `prompt create`, `get` and `update` diff --git a/openspec/changes/export-import-prompts-and-assistants/references/langsmith-sdk.md b/openspec/changes/export-import-prompts-and-assistants/references/langsmith-sdk.md index a62304c21..78e10390f 100644 --- a/openspec/changes/export-import-prompts-and-assistants/references/langsmith-sdk.md +++ b/openspec/changes/export-import-prompts-and-assistants/references/langsmith-sdk.md @@ -6,7 +6,7 @@ Read at [langsmith-sdk 47c553ca](https://github.com/langchain-ai/langsmith-sdk/t With the model included, the manifest in the [integration fixture `prompt_with_model`](https://github.com/langchain-ai/langsmith-sdk/blob/47c553caf08c1989e2be3ff1886da3eb801f709f/python/tests/integration_tests/test_prompts.py#L48) is a constructor with the `id` `["langsmith", "playground", "PromptPlayground"]`, which LangChain loads as a `RunnableSequence`. Its `first` step is the `ChatPromptTemplate` or `StructuredPrompt` (with `schema_`), and its `last` step is a `RunnableBinding` around the chat model constructor and its settings. -The chat model refers to its API key as a secret reference, `{"lc": 1, "type": "secret", "id": ["OPENAI_API_KEY"]}`. [`_manifest_has_secrets`](https://github.com/langchain-ai/langsmith-sdk/blob/47c553caf08c1989e2be3ff1886da3eb801f709f/python/langsmith/client.py#L434) finds these by matching objects whose keys are exactly `lc`, `type` and `id`. +The chat model refers to its API key as a secret reference, `{"lc": 1, "type": "secret", "id": ["OPENAI_API_KEY"]}`. [`_manifest_has_secrets`](https://github.com/langchain-ai/langsmith-sdk/blob/47c553caf08c1989e2be3ff1886da3eb801f709f/python/langsmith/client.py#L434) finds these by matching objects whose keys are exactly `lc`, `type` and `id`. [`pull_prompt`](https://github.com/langchain-ai/langsmith-sdk/blob/47c553caf08c1989e2be3ff1886da3eb801f709f/python/langsmith/client.py#L10228) defaults `secrets_from_env` to `False`, and loading a manifest that holds a secret reference raises unless the caller passes a `secrets` map or sets `secrets_from_env=True`. [`create_commit`](https://github.com/langchain-ai/langsmith-sdk/blob/47c553caf08c1989e2be3ff1886da3eb801f709f/python/langsmith/client.py#L9915) resolves a missing parent to the latest commit, posts `{parent_commit, manifest, description}`, then posts each tag to `/repos/{owner}/{repo}/tags`. [`push_prompt`](https://github.com/langchain-ai/langsmith-sdk/blob/47c553caf08c1989e2be3ff1886da3eb801f709f/python/langsmith/client.py#L10309) creates the repo first when it does not exist. diff --git a/openspec/changes/export-import-prompts-and-assistants/tasks.md b/openspec/changes/export-import-prompts-and-assistants/tasks.md index 0e31c7cbb..2a733fac5 100644 --- a/openspec/changes/export-import-prompts-and-assistants/tasks.md +++ b/openspec/changes/export-import-prompts-and-assistants/tasks.md @@ -42,7 +42,7 @@ Each numbered group is one pull request, in the order listed, except task 2.1. T - [ ] 7.1 Add `langstar prompt import`, which exits with status 1 and names `--workspace-id` when the flag is missing, runs the secret and saved model configuration checks of `prompt verify` from 6.2 before it writes, creates the repo, the commit and its tags, then reads the commit back; verify with integration tests that export from the test workspace, import into the destination workspace, and compare the full source and destination commits through the SDK with the model included, the commit description among them, and the repo's description, readme, tags and visibility, for every prompt shape from 5.1, that a file whose `requires.secrets` names a secret the destination workspace lacks creates no prompt, names the secret and exits with status 3, and a CLI test that runs import without `--workspace-id` while `LANGSMITH_WORKSPACE_ID` is set and asserts that it exits with status 1, names the missing flag and creates no prompt - [ ] 7.2 Add `--update`, `--dry-run`, `--record` and the destination record; verify with integration tests that a repeated import creates no commit, that a changed destination is refused without `--update`, that an import after someone pushed a commit to the destination since the recorded import says the destination changed since the last import and exits with status 3, that `--update` adds one commit with the right parent and the file's commit description, that `--update` against a destination whose repo description, readme, tags or visibility differ while the manifest matches updates the repo and adds no commit, that `--update` against a destination whose `prod` tag points at an earlier commit while the manifest matches moves the tag and adds no commit, that `--dry-run` lists the move, and that `--dry-run` writes nothing -- [ ] 7.3 Add the structured output check as a mise task that pulls the imported prompt with `include_model=True`, invokes it and validates the result against the schema; verify by running it from the import integration test for both methods +- [ ] 7.3 Add the structured output check as a mise task that pulls the imported prompt with `include_model=True` and a `secrets` map holding the model provider key read from the environment, invokes it and validates the result against the schema; verify by running it from the import integration test for both methods - [ ] 7.4 Document prompt import, the workspace scope of prompts and setting up destination credentials; verify that each documented command runs as written ## 8. assistant export From f7a0fdf6a63019ccff78c6654e43b182b2f639d7 Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Tue, 13 Oct 2026 19:40:20 -0400 Subject: [PATCH 15/90] =?UTF-8?q?=F0=9F=93=9A=20docs(openspec):=20record?= =?UTF-8?q?=20a=20new=20assistant's=20ID=20before=20reading=20it=20back?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit An assistant's record is the only link from the file to its copy, so assistant import writes the new ID before the read-back comparison. An import whose read-back finds a difference then leaves the link, and the next import finds that assistant instead of creating a duplicate. #807 Export, import, compare, and verify prompts and assistants across LangSmith environments Co-Authored-By: Claude Opus 5.5 --- .../changes/export-import-prompts-and-assistants/design.md | 2 +- .../specs/assistant/spec.md | 3 ++- openspec/changes/export-import-prompts-and-assistants/tasks.md | 2 +- 3 files changed, 4 insertions(+), 3 deletions(-) diff --git a/openspec/changes/export-import-prompts-and-assistants/design.md b/openspec/changes/export-import-prompts-and-assistants/design.md index 31f7cde46..082103eaf 100644 --- a/openspec/changes/export-import-prompts-and-assistants/design.md +++ b/openspec/changes/export-import-prompts-and-assistants/design.md @@ -57,7 +57,7 @@ Export reads the commit with `include=model`, the form the Go SDK uses (`referen ### Import records what it created in a separate file -Import writes the destination's IDs to `.destinations.json`, keyed by destination workspace ID for prompts and by deployment ID for assistants. The export file stays the same after any number of imports, and a fresh export never wipes out the record of earlier imports. A repeated import reads the record to find the copy it made. For a prompt, langstar finds the copy by its handle, and compares the commit hash in the record with the destination's latest commit to tell whether someone changed the prompt since the last import. For an assistant, whose ID differs on every deployment, the record is the only link from the file to its copy. +Import writes the destination's IDs to `.destinations.json`, keyed by destination workspace ID for prompts and by deployment ID for assistants. The export file stays the same after any number of imports, and a fresh export never wipes out the record of earlier imports. A repeated import reads the record to find the copy it made. For a prompt, langstar finds the copy by its handle, and compares the commit hash in the record with the destination's latest commit to tell whether someone changed the prompt since the last import. For an assistant, whose ID differs on every deployment, the record is the only link from the file to its copy. Assistant import therefore writes the new ID to the record before it reads the assistant back, so an import whose read-back finds a difference still leaves the link, and the next import finds that assistant instead of creating a second one. ### Import changes an existing resource only with --update diff --git a/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md b/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md index 988b4c366..b5d0f045c 100644 --- a/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md +++ b/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md @@ -144,12 +144,13 @@ When the destination record holds an assistant ID for the deployment, `langstar ### Requirement: Read the assistant back after import -After writing, `langstar assistant import` SHALL read the assistant back from the deployment and compare it with the file. When they differ it SHALL list the differences and exit with status 3. +After writing, `langstar assistant import` SHALL record the assistant ID in the destination record, then read the assistant back from the deployment and compare it with the file. When they differ it SHALL list the differences and exit with status 3, and the record SHALL keep the ID so that the next import finds that assistant instead of creating another. #### Scenario: Deployment drops a configuration key - **WHEN** the assistant read back lacks a configuration key the file holds - **THEN** langstar lists that key as a difference and exits with status 3 +- **AND** the destination record holds the new assistant ID, so importing the file again creates no assistant ### Requirement: Compare an assistant with its export file diff --git a/openspec/changes/export-import-prompts-and-assistants/tasks.md b/openspec/changes/export-import-prompts-and-assistants/tasks.md index 2a733fac5..c0bce2876 100644 --- a/openspec/changes/export-import-prompts-and-assistants/tasks.md +++ b/openspec/changes/export-import-prompts-and-assistants/tasks.md @@ -58,7 +58,7 @@ Each numbered group is one pull request, in the order listed, except task 2.1. T ## 10. assistant import -- [ ] 10.1 Add `langstar assistant import` with `--workspace-id`, the graph check and prompt check of `assistant verify` from 9.2 run before it writes, creation, read-back, the destination record and `--record`; verify with an integration test that creates a tagged prompt in the test workspace, imports an assistant that refers to it into the destination deployment and checks name, graph ID, configuration, context, metadata and prompt references through the Agent Server API, that `--record ` writes the destination record to that path, and that a file whose graph ID the deployment lacks and a file with a missing prompt tag each create no assistant and exit with status 3 +- [ ] 10.1 Add `langstar assistant import` with `--workspace-id`, the graph check and prompt check of `assistant verify` from 9.2 run before it writes, creation, read-back, the destination record and `--record`; verify with an integration test that creates a tagged prompt in the test workspace, imports an assistant that refers to it into the destination deployment and checks name, graph ID, configuration, context, metadata and prompt references through the Agent Server API, that `--record ` writes the destination record to that path, and that a file whose graph ID the deployment lacks and a file with a missing prompt tag each create no assistant and exit with status 3, and an httpmock test in which the assistant read back differs from the file, asserting that import exits with status 3, that the destination record holds the new assistant ID, and that a second import creates no assistant - [ ] 10.2 Add `--update`, `--assistant-id` and `--dry-run`; verify with integration tests that a repeated import creates no assistant, that `--update` changes the recorded one, its graph ID among the fields, that `--update --assistant-id ` naming an assistant other than the recorded one updates that assistant, leaves the recorded one unchanged and writes the new ID to the destination record, that `--update` of an assistant with metadata `{"kept": 1, "stale": 2}` from a file with `{"kept": 1}` leaves `{"kept": 1}` or reports `/metadata/stale` and exits with status 3, whichever task 1.2 found the server does, that `--update` of an assistant with a description and context from a file with neither clears both or reports `/description` and `/context` and exits with status 3, whichever task 1.2 found the server does, that a deleted recorded assistant exits with status 1, and that `--dry-run` creates no assistant, changes none and leaves the destination record as it was - [ ] 10.3 Add an end-to-end test that imports prompts into the destination workspace, then imports an assistant that refers to them onto the destination deployment and runs `assistant verify`, passing `--workspace-id` set to the destination workspace to both assistant commands; verify that it passes, and that verify fails after deleting one prompt tag in the destination workspace - [ ] 10.4 Document assistant import, the deployment scope of assistants, and how export and import relate to the `create`, `get` and `update` commands of [#728 Implement CRUD command structure (create, get, update) for prompt commands](https://github.com/codekiln/langstar/issues/728); verify that each documented command runs as written From 86a62f4ba4ac4c7be2ab736a4b590c4c6a3e42b0 Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Tue, 13 Oct 2026 19:41:41 -0400 Subject: [PATCH 16/90] =?UTF-8?q?=F0=9F=93=9A=20docs(openspec):=20reject?= =?UTF-8?q?=20an=20export=20file=20of=20another=20kind=20or=20format=20ver?= =?UTF-8?q?sion?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Import, compare and verify read only a file whose format_version is 1 and whose kind names their own resource, and exit with status 1 naming the field otherwise. Task 4.1 adds the check with unit tests, and task 6.1 adds CLI tests for both cases. #807 Export, import, compare, and verify prompts and assistants across LangSmith environments Co-Authored-By: Claude Opus 5.5 --- .../design.md | 2 ++ .../specs/assistant/spec.md | 10 ++++++++++ .../specs/prompt/spec.md | 16 ++++++++++++++++ .../tasks.md | 4 ++-- 4 files changed, 30 insertions(+), 2 deletions(-) diff --git a/openspec/changes/export-import-prompts-and-assistants/design.md b/openspec/changes/export-import-prompts-and-assistants/design.md index 082103eaf..795a4ad34 100644 --- a/openspec/changes/export-import-prompts-and-assistants/design.md +++ b/openspec/changes/export-import-prompts-and-assistants/design.md @@ -51,6 +51,8 @@ An assistant export has `kind: "assistant"`, a `source` with the deployment, ass The file holds no export time and langstar sorts its keys, so exporting an unchanged resource leaves Git with nothing to commit. +Import, compare and verify read only files whose `format_version` is 1 and whose `kind` names their own resource. Any other file makes them exit with status 1 and name the field, so a file from a newer langstar, or an assistant file given to a prompt command, is never read as something it is not. + ### Export asks LangSmith for the model Export reads the commit with `include=model`, the form the Go SDK uses (`references/langsmith-go.md`). The Python SDK sends the older `include_model=true`, and the LangSmith OpenAPI spec accepts both. With the model included, the manifest wraps the prompt and the chat model with its settings in one constructor, and the response also carries `model_config` and `model_provider` beside the manifest (`references/langsmith-sdk.md`). No field of the create-commit request holds the model, so the model reaches the destination inside the manifest. [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8) pushes that manifest to a second workspace, records whether LangSmith returns the same model, and records which outer `id` the live API returns. diff --git a/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md b/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md index b5d0f045c..dd9a218bb 100644 --- a/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md +++ b/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md @@ -176,6 +176,16 @@ After writing, `langstar assistant import` SHALL record the assistant ID in the - **AND** the workspace has no `support-system` prompt with a `prod` tag - **THEN** langstar reports `support-system:prod` as missing and exits with status 3 +### Requirement: Read only an assistant export file this version understands + +`assistant import`, `assistant compare` and `assistant verify` SHALL read a file only when its `kind` is `assistant` and its `format_version` is 1. For any other file they SHALL write nothing, exit with status 1 and name the field and the value they found. + +#### Scenario: Prompt file given to an assistant command + +- **WHEN** a user runs `langstar assistant import my-prompt.json --deployment my-agent-prod` +- **AND** `my-prompt.json` holds `"kind": "prompt"` +- **THEN** langstar creates no assistant, exits with status 1 and names `kind` and the value `prompt` + ### Requirement: Assistant compare, verify and import use the prompt commands' exit statuses and JSON report `assistant compare`, `assistant verify` and `assistant import` SHALL use the exit statuses and the JSON report shape of `prompt compare`, `prompt verify` and `prompt import`. diff --git a/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md index d3f2786cd..ad855907f 100644 --- a/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md +++ b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md @@ -205,6 +205,22 @@ After writing, `langstar prompt import` SHALL read the destination commit back w - **AND** every secret in `requires.secrets` exists and the prompt matches the file - **THEN** langstar notes that no saved model configuration matches `ChatOpenAI` and exits with status 0 +### Requirement: Read only a prompt export file this version understands + +`prompt import`, `prompt compare` and `prompt verify` SHALL read a file only when its `kind` is `prompt` and its `format_version` is 1. For any other file they SHALL write nothing, exit with status 1 and name the field and the value they found. + +#### Scenario: File from a newer format + +- **WHEN** a user runs `langstar prompt compare my-prompt.json` +- **AND** `my-prompt.json` holds `"format_version": 2` +- **THEN** langstar exits with status 1 and an error naming `format_version` and the value 2 + +#### Scenario: Assistant file given to a prompt command + +- **WHEN** a user runs `langstar prompt import support-bot.json --workspace-id ` +- **AND** `support-bot.json` holds `"kind": "assistant"` +- **THEN** langstar creates no prompt, exits with status 1 and names `kind` and the value `assistant` + ### Requirement: Exit with 0 for a match, 3 for a difference and 1 for an error `prompt compare`, `prompt verify` and `prompt import` SHALL exit with status 0 when the prompt matches and every check passes, 3 when they find a difference or a failed check, and 1 on any other error. A usage error that clap rejects, such as a mistyped flag, SHALL keep clap's status 2. With `-f json` each SHALL print one JSON object with `status`, `differences` and `checks`. diff --git a/openspec/changes/export-import-prompts-and-assistants/tasks.md b/openspec/changes/export-import-prompts-and-assistants/tasks.md index c0bce2876..161f12373 100644 --- a/openspec/changes/export-import-prompts-and-assistants/tasks.md +++ b/openspec/changes/export-import-prompts-and-assistants/tasks.md @@ -22,7 +22,7 @@ Each numbered group is one pull request, in the order listed, except task 2.1. T ## 4. Shared code for the prompt and assistant commands: the export file, the comparison and the report -- [ ] 4.1 Add the export file types, with sorted keys and no export time, and the destination record file, to a new module in `cli/src/`; verify with unit tests that build equivalent nested values with keys inserted in different orders, serialize each, and assert identical text with keys sorted at every level, and that updating one destination leaves the others unchanged +- [ ] 4.1 Add the export file types, with sorted keys, no export time and a reader that accepts only `format_version` 1 and the expected `kind`, and the destination record file, to a new module in `cli/src/`; verify with unit tests that build equivalent nested values with keys inserted in different orders, serialize each, and assert identical text with keys sorted at every level, and that updating one destination leaves the others unchanged, and that reading a file whose `format_version` is 2, or whose `kind` names the other resource, returns an error naming the field and its value - [ ] 4.2 Add the JSON comparison that drops server-set fields and reports differences by JSON Pointer; verify with unit tests on nested objects, arrays, missing keys and ignored fields - [ ] 4.3 Add the shared result report and the exit statuses 0, 3 and 1, printing JSON to stdout and progress to stderr, and leave clap's status 2 for usage errors; verify with unit tests of the report and of the exit status each result gives - [ ] 4.4 Add the secret reference walk that lists secret names from a manifest; verify with unit tests on a copy of the `prompt_with_model` manifest from the LangSmith Python SDK's integration tests, which `references/langsmith-sdk.md` links, and on a manifest with no secret reference @@ -34,7 +34,7 @@ Each numbered group is one pull request, in the order listed, except task 2.1. T ## 6. prompt compare and verify -- [ ] 6.1 Add `langstar prompt compare`; verify with integration tests that export a prompt, change its model temperature through the SDK, and assert the reported path, both values and exit status 3, and that a changed commit description and a changed repo readme are each reported the same way, and CLI tests that `langstar -f json prompt compare` prints one JSON object with `status`, `differences` and `checks` to stdout and its progress to stderr, and that `prompt compare` with a mistyped flag exits with status 2 +- [ ] 6.1 Add `langstar prompt compare`; verify with integration tests that export a prompt, change its model temperature through the SDK, and assert the reported path, both values and exit status 3, and that a changed commit description and a changed repo readme are each reported the same way, and CLI tests that `langstar -f json prompt compare` prints one JSON object with `status`, `differences` and `checks` to stdout and its progress to stderr, and that `prompt compare` with a mistyped flag exits with status 2, and that `prompt compare` of a file holding `"format_version": 2` and `prompt import` of an assistant export each exit with status 1 and name the field - [ ] 6.2 Add `langstar prompt verify` with the secret check and the saved model configuration note, matching `requires.model_provider` against the class name at the end of each configuration's settings `id`; verify with integration tests against a workspace that lacks a listed secret and one that has it, and a test that a missing saved model configuration is reported and leaves the exit status at 0 - [ ] 6.3 Document compare and verify; verify that each documented command runs as written From 9d9ca137846b10da76b58ccfdd48c09847824e6f Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Tue, 13 Oct 2026 19:50:19 -0400 Subject: [PATCH 17/90] =?UTF-8?q?=F0=9F=93=9A=20docs(openspec):=20record?= =?UTF-8?q?=20the=20prompt=20commit=20before=20reading=20it=20back,=20and?= =?UTF-8?q?=20test=20both=20read-back=20failures?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Prompt import now records the commit hash before the read-back, as assistant import does, so the next import does not report langstar's own commit as someone else's change. Task 7.2 adds an httpmock test for a field LangSmith drops, task 10.1 a CLI test for a prompt export given to assistant import, and design.md splits the verify paragraph into the secret check and the model configuration note. #807 Export, import, compare, and verify prompts and assistants across LangSmith environments Co-Authored-By: Claude Opus 5.5 --- .../export-import-prompts-and-assistants/design.md | 8 +++++--- .../specs/prompt/spec.md | 5 +++-- .../changes/export-import-prompts-and-assistants/tasks.md | 4 ++-- 3 files changed, 10 insertions(+), 7 deletions(-) diff --git a/openspec/changes/export-import-prompts-and-assistants/design.md b/openspec/changes/export-import-prompts-and-assistants/design.md index 795a4ad34..7e294fabd 100644 --- a/openspec/changes/export-import-prompts-and-assistants/design.md +++ b/openspec/changes/export-import-prompts-and-assistants/design.md @@ -51,7 +51,7 @@ An assistant export has `kind: "assistant"`, a `source` with the deployment, ass The file holds no export time and langstar sorts its keys, so exporting an unchanged resource leaves Git with nothing to commit. -Import, compare and verify read only files whose `format_version` is 1 and whose `kind` names their own resource. Any other file makes them exit with status 1 and name the field, so a file from a newer langstar, or an assistant file given to a prompt command, is never read as something it is not. +Import, compare and verify read a file only when its `format_version` is 1 and its `kind` matches the command: `prompt` for the prompt commands and `assistant` for the assistant commands. For any other file the command writes nothing, exits with status 1 and names the field and the value it found, so a file from a newer langstar, or an assistant file given to `prompt import`, stops the command before it changes anything. ### Export asks LangSmith for the model @@ -59,7 +59,7 @@ Export reads the commit with `include=model`, the form the Go SDK uses (`referen ### Import records what it created in a separate file -Import writes the destination's IDs to `.destinations.json`, keyed by destination workspace ID for prompts and by deployment ID for assistants. The export file stays the same after any number of imports, and a fresh export never wipes out the record of earlier imports. A repeated import reads the record to find the copy it made. For a prompt, langstar finds the copy by its handle, and compares the commit hash in the record with the destination's latest commit to tell whether someone changed the prompt since the last import. For an assistant, whose ID differs on every deployment, the record is the only link from the file to its copy. Assistant import therefore writes the new ID to the record before it reads the assistant back, so an import whose read-back finds a difference still leaves the link, and the next import finds that assistant instead of creating a second one. +Import writes the destination's IDs to `.destinations.json`, keyed by destination workspace ID for prompts and by deployment ID for assistants. The export file stays the same after any number of imports, and a fresh export never wipes out the record of earlier imports. A repeated import reads the record to find the copy it made. For a prompt, langstar finds the copy by its handle, and compares the commit hash in the record with the destination's latest commit to tell whether someone changed the prompt since the last import. For an assistant, whose ID differs on every deployment, the record is the only link from the file to its copy. Import writes the new commit hash or assistant ID to the record before it reads the copy back, so an import whose read-back finds a difference still records what it wrote. The next prompt import then compares the destination's latest commit with the commit langstar wrote, and reports a change only when someone else pushed one; the next assistant import finds that assistant instead of creating a second one. ### Import changes an existing resource only with --update @@ -75,7 +75,9 @@ Compare reads the destination, removes the fields the server sets (IDs, version ### Verify checks what the copy needs to run -Prompt verify looks up each name in `requires.secrets` among the workspace's secrets (`GET /api/v1/workspaces/current/secrets`, which `langstar secrets list` already calls). It also looks for a saved model configuration (`GET /api/v1/playground-settings`, which `langstar model-config list` calls) whose settings `id` ends in the class name that `requires.model_provider` holds, such as `ChatOpenAI`. Export reads that class name from the last element of the chat model constructor's `id` in the manifest, such as `["langchain", "chat_models", "openai", "ChatOpenAI"]`. LangSmith also returns a `model_provider` field beside the manifest, and the file keeps it as `commit.model_provider`. No SDK or spec shows what that field holds, so [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8) records it. The prompt runs without a saved model configuration, because its manifest carries the model, so verify reports a missing one as a note and leaves the exit status alone; [#807 Export, import, compare, and verify prompts and assistants across LangSmith environments](https://github.com/codekiln/langstar/issues/807) asks for the report, and [Should prompt verify report a missing saved model configuration?](#7---should-prompt-verify-report-a-missing-saved-model-configuration) asks whether to keep it. Verify prints the name of each missing secret and no secret value. Langstar finds secret names by walking the manifest for LangChain secret references, `{"lc": 1, "type": "secret", "id": ["OPENAI_API_KEY"]}`, the shape the Python SDK's `_manifest_has_secrets` matches. +Prompt verify checks that the workspace holds every secret the prompt's model reads. Export lists those names in `requires.secrets` by walking the manifest for LangChain secret references, `{"lc": 1, "type": "secret", "id": ["OPENAI_API_KEY"]}`, the shape the Python SDK's `_manifest_has_secrets` matches. Verify looks each name up among the workspace's secrets (`GET /api/v1/workspaces/current/secrets`, which `langstar secrets list` already calls) and prints the name of each missing secret and no secret value. + +Prompt verify also notes when the workspace has no saved model configuration for the prompt's model, and leaves the exit status alone, because the prompt runs with the model its manifest carries. Export takes the model's class name, such as `ChatOpenAI`, from the last element of the chat model constructor's `id` in the manifest, such as `["langchain", "chat_models", "openai", "ChatOpenAI"]`, and stores it in `requires.model_provider`. Verify looks for a saved model configuration (`GET /api/v1/playground-settings`, which `langstar model-config list` calls) whose settings `id` ends in that class name. LangSmith also returns a `model_provider` field beside the manifest, which the file keeps as `commit.model_provider`; no SDK or spec shows what it holds, so [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8) records it. [#807 Export, import, compare, and verify prompts and assistants across LangSmith environments](https://github.com/codekiln/langstar/issues/807) asks for the note, and [Should prompt verify report a missing saved model configuration?](#7---should-prompt-verify-report-a-missing-saved-model-configuration) asks whether to keep it. Assistant verify checks the graph with `POST /assistants/search` filtered to the graph ID; every graph on a deployment has a default assistant, so an empty result means the deployment does not serve the graph. It checks each prompt in `requires.prompts`, such as `support-system:prod`, by asking LangSmith for the commit that tag points at (`GET /api/v1/commits/{owner}/{repo}/{tag}`, the request `langstar prompt pull --commit prod` already sends). A 404 means the workspace lacks the prompt or its tag. diff --git a/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md index ad855907f..5b16a955e 100644 --- a/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md +++ b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md @@ -132,7 +132,7 @@ When a commit tag in the file is missing in the destination or points at a diffe ### Requirement: Record the destination commit -After a successful import, langstar SHALL record the destination workspace, handle and commit hash in a destination record file, `.destinations.json` next to the export unless `--record ` names another. Each import SHALL add or replace only the entry for its own destination workspace. +After import writes a commit, langstar SHALL record the destination workspace, handle and commit hash in a destination record file, `.destinations.json` next to the export unless `--record ` names another, before it reads the commit back. Each import SHALL add or replace only the entry for its own destination workspace. #### Scenario: Import into two workspaces @@ -162,12 +162,13 @@ When the destination record holds a commit for the destination workspace and the ### Requirement: Read the prompt back after import -After writing, `langstar prompt import` SHALL read the destination commit back with the model included and compare it with the file. When they differ it SHALL list the differences and exit with status 3. +After writing, `langstar prompt import` SHALL record the commit hash in the destination record, then read the destination commit back with the model included and compare it with the file. When they differ it SHALL list the differences and exit with status 3, and the record SHALL keep the hash so that the next import does not report the commit langstar wrote as someone else's change. #### Scenario: LangSmith drops a field - **WHEN** the destination commit read back after import lacks a field the file holds - **THEN** langstar lists that field as a difference and exits with status 3 +- **AND** the destination record holds the new commit hash ### Requirement: Compare a prompt with its export file diff --git a/openspec/changes/export-import-prompts-and-assistants/tasks.md b/openspec/changes/export-import-prompts-and-assistants/tasks.md index 161f12373..2c9632cca 100644 --- a/openspec/changes/export-import-prompts-and-assistants/tasks.md +++ b/openspec/changes/export-import-prompts-and-assistants/tasks.md @@ -41,7 +41,7 @@ Each numbered group is one pull request, in the order listed, except task 2.1. T ## 7. prompt import - [ ] 7.1 Add `langstar prompt import`, which exits with status 1 and names `--workspace-id` when the flag is missing, runs the secret and saved model configuration checks of `prompt verify` from 6.2 before it writes, creates the repo, the commit and its tags, then reads the commit back; verify with integration tests that export from the test workspace, import into the destination workspace, and compare the full source and destination commits through the SDK with the model included, the commit description among them, and the repo's description, readme, tags and visibility, for every prompt shape from 5.1, that a file whose `requires.secrets` names a secret the destination workspace lacks creates no prompt, names the secret and exits with status 3, and a CLI test that runs import without `--workspace-id` while `LANGSMITH_WORKSPACE_ID` is set and asserts that it exits with status 1, names the missing flag and creates no prompt -- [ ] 7.2 Add `--update`, `--dry-run`, `--record` and the destination record; verify with integration tests that a repeated import creates no commit, that a changed destination is refused without `--update`, that an import after someone pushed a commit to the destination since the recorded import says the destination changed since the last import and exits with status 3, that `--update` adds one commit with the right parent and the file's commit description, that `--update` against a destination whose repo description, readme, tags or visibility differ while the manifest matches updates the repo and adds no commit, that `--update` against a destination whose `prod` tag points at an earlier commit while the manifest matches moves the tag and adds no commit, that `--dry-run` lists the move, and that `--dry-run` writes nothing +- [ ] 7.2 Add `--update`, `--dry-run`, `--record` and the destination record; verify with integration tests that a repeated import creates no commit, that a changed destination is refused without `--update`, that an import after someone pushed a commit to the destination since the recorded import says the destination changed since the last import and exits with status 3, that `--update` adds one commit with the right parent and the file's commit description, that `--update` against a destination whose repo description, readme, tags or visibility differ while the manifest matches updates the repo and adds no commit, that `--update` against a destination whose `prod` tag points at an earlier commit while the manifest matches moves the tag and adds no commit, that `--dry-run` lists the move, and that `--dry-run` writes nothing, and an httpmock test in which the commit read back lacks a field the file holds, asserting that import names that field's JSON Pointer path, exits with status 3, and leaves the new commit hash in the destination record - [ ] 7.3 Add the structured output check as a mise task that pulls the imported prompt with `include_model=True` and a `secrets` map holding the model provider key read from the environment, invokes it and validates the result against the schema; verify by running it from the import integration test for both methods - [ ] 7.4 Document prompt import, the workspace scope of prompts and setting up destination credentials; verify that each documented command runs as written @@ -58,7 +58,7 @@ Each numbered group is one pull request, in the order listed, except task 2.1. T ## 10. assistant import -- [ ] 10.1 Add `langstar assistant import` with `--workspace-id`, the graph check and prompt check of `assistant verify` from 9.2 run before it writes, creation, read-back, the destination record and `--record`; verify with an integration test that creates a tagged prompt in the test workspace, imports an assistant that refers to it into the destination deployment and checks name, graph ID, configuration, context, metadata and prompt references through the Agent Server API, that `--record ` writes the destination record to that path, and that a file whose graph ID the deployment lacks and a file with a missing prompt tag each create no assistant and exit with status 3, and an httpmock test in which the assistant read back differs from the file, asserting that import exits with status 3, that the destination record holds the new assistant ID, and that a second import creates no assistant +- [ ] 10.1 Add `langstar assistant import` with `--workspace-id`, the graph check and prompt check of `assistant verify` from 9.2 run before it writes, creation, read-back, the destination record and `--record`; verify with an integration test that creates a tagged prompt in the test workspace, imports an assistant that refers to it into the destination deployment and checks name, graph ID, configuration, context, metadata and prompt references through the Agent Server API, that `--record ` writes the destination record to that path, and that a file whose graph ID the deployment lacks and a file with a missing prompt tag each create no assistant and exit with status 3, and an httpmock test in which the assistant read back differs from the file, asserting that import exits with status 3, that the destination record holds the new assistant ID, and that a second import creates no assistant, and a CLI test that `assistant import` of a prompt export exits with status 1, names `kind` and the value `prompt`, and creates no assistant - [ ] 10.2 Add `--update`, `--assistant-id` and `--dry-run`; verify with integration tests that a repeated import creates no assistant, that `--update` changes the recorded one, its graph ID among the fields, that `--update --assistant-id ` naming an assistant other than the recorded one updates that assistant, leaves the recorded one unchanged and writes the new ID to the destination record, that `--update` of an assistant with metadata `{"kept": 1, "stale": 2}` from a file with `{"kept": 1}` leaves `{"kept": 1}` or reports `/metadata/stale` and exits with status 3, whichever task 1.2 found the server does, that `--update` of an assistant with a description and context from a file with neither clears both or reports `/description` and `/context` and exits with status 3, whichever task 1.2 found the server does, that a deleted recorded assistant exits with status 1, and that `--dry-run` creates no assistant, changes none and leaves the destination record as it was - [ ] 10.3 Add an end-to-end test that imports prompts into the destination workspace, then imports an assistant that refers to them onto the destination deployment and runs `assistant verify`, passing `--workspace-id` set to the destination workspace to both assistant commands; verify that it passes, and that verify fails after deleting one prompt tag in the destination workspace - [ ] 10.4 Document assistant import, the deployment scope of assistants, and how export and import relate to the `create`, `get` and `update` commands of [#728 Implement CRUD command structure (create, get, update) for prompt commands](https://github.com/codekiln/langstar/issues/728); verify that each documented command runs as written From b0564cf0ae1ba3b7064ccbc00b685a23fb47dc03 Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Tue, 13 Oct 2026 20:05:29 -0400 Subject: [PATCH 18/90] =?UTF-8?q?=F0=9F=93=9A=20docs(openspec):=20test=20a?= =?UTF-8?q?n=20assistant=20file=20given=20to=20prompt=20import=20with=20im?= =?UTF-8?q?port,=20and=20name=20the=20format-version=20requirement?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit #807 Export, import, compare, and verify prompts and assistants across LangSmith environments Co-Authored-By: Claude Opus 5.5 --- .../changes/export-import-prompts-and-assistants/design.md | 2 +- .../specs/assistant/spec.md | 2 +- .../specs/prompt/spec.md | 4 ++-- .../changes/export-import-prompts-and-assistants/tasks.md | 6 +++--- 4 files changed, 7 insertions(+), 7 deletions(-) diff --git a/openspec/changes/export-import-prompts-and-assistants/design.md b/openspec/changes/export-import-prompts-and-assistants/design.md index 7e294fabd..511fb5aff 100644 --- a/openspec/changes/export-import-prompts-and-assistants/design.md +++ b/openspec/changes/export-import-prompts-and-assistants/design.md @@ -83,7 +83,7 @@ Assistant verify checks the graph with `POST /assistants/search` filtered to the ### Import checks the destination's secrets, graph and prompts before it writes -Before it writes anything, import runs the dependency checks verify runs against the destination, because [#807 Export, import, compare, and verify prompts and assistants across LangSmith environments](https://github.com/codekiln/langstar/issues/807) asks import to check the destination's dependencies. It skips verify's comparison, since the destination may have no copy yet. Prompt import stops when the destination workspace lacks a secret in `requires.secrets`, and assistant import stops when the deployment does not serve the file's graph or the workspace, from `--workspace-id` or the configured workspace, lacks a prompt in `requires.prompts`. Each names what is missing, writes nothing and exits with status 3, and `--dry-run` reports the same failed checks. Import therefore writes a copy only when the destination has the secrets, graph and prompts that copy needs to run. A user therefore has to create the destination workspace's secrets before importing a prompt, and import the prompts before importing an assistant that refers to them. [Should a failed dependency check stop import before it writes?](#8---should-a-failed-dependency-check-stop-import-before-it-writes) asks whether import should write anyway and report the failed checks afterwards. +Before it writes anything, import checks that the destination has the secrets, the graph and the prompts the copy needs. These are the checks `verify` makes, and import runs them because [#807 Export, import, compare, and verify prompts and assistants across LangSmith environments](https://github.com/codekiln/langstar/issues/807) asks import to check the destination's dependencies. Import skips verify's comparison, since the destination may have no copy yet. Prompt import stops when the destination workspace lacks a secret in `requires.secrets`, and assistant import stops when the deployment does not serve the file's graph or the workspace, from `--workspace-id` or the configured workspace, lacks a prompt in `requires.prompts`. Each names what is missing, writes nothing and exits with status 3, and `--dry-run` reports the same failed checks. A user therefore creates the destination workspace's secrets before importing a prompt, and imports the prompts before importing an assistant that refers to them. [Should a failed dependency check stop import before it writes?](#8---should-a-failed-dependency-check-stop-import-before-it-writes) asks whether import should write anyway and report the failed checks afterwards. ### The user names an assistant's prompt references diff --git a/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md b/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md index dd9a218bb..295d01b6f 100644 --- a/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md +++ b/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md @@ -176,7 +176,7 @@ After writing, `langstar assistant import` SHALL record the assistant ID in the - **AND** the workspace has no `support-system` prompt with a `prod` tag - **THEN** langstar reports `support-system:prod` as missing and exits with status 3 -### Requirement: Read only an assistant export file this version understands +### Requirement: Reject an export file of another kind or format version `assistant import`, `assistant compare` and `assistant verify` SHALL read a file only when its `kind` is `assistant` and its `format_version` is 1. For any other file they SHALL write nothing, exit with status 1 and name the field and the value they found. diff --git a/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md index 5b16a955e..324ed520f 100644 --- a/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md +++ b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md @@ -162,7 +162,7 @@ When the destination record holds a commit for the destination workspace and the ### Requirement: Read the prompt back after import -After writing, `langstar prompt import` SHALL record the commit hash in the destination record, then read the destination commit back with the model included and compare it with the file. When they differ it SHALL list the differences and exit with status 3, and the record SHALL keep the hash so that the next import does not report the commit langstar wrote as someone else's change. +After writing, `langstar prompt import` SHALL record the commit hash in the destination record, then read the destination commit back with the model included and compare it with the file. When they differ it SHALL list the differences and exit with status 3, and the record SHALL keep the hash, so that the next import reports a change only when someone pushed a commit after the one langstar wrote. #### Scenario: LangSmith drops a field @@ -206,7 +206,7 @@ After writing, `langstar prompt import` SHALL record the commit hash in the dest - **AND** every secret in `requires.secrets` exists and the prompt matches the file - **THEN** langstar notes that no saved model configuration matches `ChatOpenAI` and exits with status 0 -### Requirement: Read only a prompt export file this version understands +### Requirement: Reject an export file of another kind or format version `prompt import`, `prompt compare` and `prompt verify` SHALL read a file only when its `kind` is `prompt` and its `format_version` is 1. For any other file they SHALL write nothing, exit with status 1 and name the field and the value they found. diff --git a/openspec/changes/export-import-prompts-and-assistants/tasks.md b/openspec/changes/export-import-prompts-and-assistants/tasks.md index 2c9632cca..9d25fc7cb 100644 --- a/openspec/changes/export-import-prompts-and-assistants/tasks.md +++ b/openspec/changes/export-import-prompts-and-assistants/tasks.md @@ -1,6 +1,6 @@ # Tasks -Each numbered group is one pull request, in the order listed, except task 2.1. Task 2.1 waits on codekiln, so it goes in a pull request of its own that lands before group 6, and the rest of group 2 goes ahead without it. +Each numbered group below is one pull request, merged in the order listed. Task 2.1, which adds the destination workspace and the model provider key, waits on codekiln to create that workspace and an OpenAI key, so it gets a pull request of its own and the rest of group 2 goes ahead without it. That pull request merges before group 6, prompt compare and verify, whose tests are the first to use the destination workspace. ## 1. Check the API round trip before building on it @@ -34,13 +34,13 @@ Each numbered group is one pull request, in the order listed, except task 2.1. T ## 6. prompt compare and verify -- [ ] 6.1 Add `langstar prompt compare`; verify with integration tests that export a prompt, change its model temperature through the SDK, and assert the reported path, both values and exit status 3, and that a changed commit description and a changed repo readme are each reported the same way, and CLI tests that `langstar -f json prompt compare` prints one JSON object with `status`, `differences` and `checks` to stdout and its progress to stderr, and that `prompt compare` with a mistyped flag exits with status 2, and that `prompt compare` of a file holding `"format_version": 2` and `prompt import` of an assistant export each exit with status 1 and name the field +- [ ] 6.1 Add `langstar prompt compare`; verify with integration tests that export a prompt, change its model temperature through the SDK, and assert the reported path, both values and exit status 3, and that a changed commit description and a changed repo readme are each reported the same way, and CLI tests that `langstar -f json prompt compare` prints one JSON object with `status`, `differences` and `checks` to stdout and its progress to stderr, and that `prompt compare` with a mistyped flag exits with status 2, and that `prompt compare` of a file holding `"format_version": 2` exits with status 1 and names `format_version` and the value 2 - [ ] 6.2 Add `langstar prompt verify` with the secret check and the saved model configuration note, matching `requires.model_provider` against the class name at the end of each configuration's settings `id`; verify with integration tests against a workspace that lacks a listed secret and one that has it, and a test that a missing saved model configuration is reported and leaves the exit status at 0 - [ ] 6.3 Document compare and verify; verify that each documented command runs as written ## 7. prompt import -- [ ] 7.1 Add `langstar prompt import`, which exits with status 1 and names `--workspace-id` when the flag is missing, runs the secret and saved model configuration checks of `prompt verify` from 6.2 before it writes, creates the repo, the commit and its tags, then reads the commit back; verify with integration tests that export from the test workspace, import into the destination workspace, and compare the full source and destination commits through the SDK with the model included, the commit description among them, and the repo's description, readme, tags and visibility, for every prompt shape from 5.1, that a file whose `requires.secrets` names a secret the destination workspace lacks creates no prompt, names the secret and exits with status 3, and a CLI test that runs import without `--workspace-id` while `LANGSMITH_WORKSPACE_ID` is set and asserts that it exits with status 1, names the missing flag and creates no prompt +- [ ] 7.1 Add `langstar prompt import`, which exits with status 1 and names `--workspace-id` when the flag is missing, runs the secret and saved model configuration checks of `prompt verify` from 6.2 before it writes, creates the repo, the commit and its tags, then reads the commit back; verify with integration tests that export from the test workspace, import into the destination workspace, and compare the full source and destination commits through the SDK with the model included, the commit description among them, and the repo's description, readme, tags and visibility, for every prompt shape from 5.1, that a file whose `requires.secrets` names a secret the destination workspace lacks creates no prompt, names the secret and exits with status 3, and a CLI test that runs import without `--workspace-id` while `LANGSMITH_WORKSPACE_ID` is set and asserts that it exits with status 1, names the missing flag and creates no prompt, and a CLI test that `prompt import` of an assistant export exits with status 1, names `kind` and the value `assistant`, and creates no prompt - [ ] 7.2 Add `--update`, `--dry-run`, `--record` and the destination record; verify with integration tests that a repeated import creates no commit, that a changed destination is refused without `--update`, that an import after someone pushed a commit to the destination since the recorded import says the destination changed since the last import and exits with status 3, that `--update` adds one commit with the right parent and the file's commit description, that `--update` against a destination whose repo description, readme, tags or visibility differ while the manifest matches updates the repo and adds no commit, that `--update` against a destination whose `prod` tag points at an earlier commit while the manifest matches moves the tag and adds no commit, that `--dry-run` lists the move, and that `--dry-run` writes nothing, and an httpmock test in which the commit read back lacks a field the file holds, asserting that import names that field's JSON Pointer path, exits with status 3, and leaves the new commit hash in the destination record - [ ] 7.3 Add the structured output check as a mise task that pulls the imported prompt with `include_model=True` and a `secrets` map holding the model provider key read from the environment, invokes it and validates the result against the schema; verify by running it from the import integration test for both methods - [ ] 7.4 Document prompt import, the workspace scope of prompts and setting up destination credentials; verify that each documented command runs as written From 0e6190417d5e9313c25c530ca1bac9cbec0ccf56 Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Tue, 13 Oct 2026 20:16:09 -0400 Subject: [PATCH 19/90] =?UTF-8?q?=F0=9F=93=9A=20docs(openspec):=20give=20t?= =?UTF-8?q?he=20destination=20record=20its=20prompt=20purpose,=20make=20de?= =?UTF-8?q?velopers=20the=20actors=20of=20task=20sentences,=20and=20drop?= =?UTF-8?q?=20the=20#728=20field=20claim?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The proposal now says the record finds an assistant's copy and tells whether someone pushed to a prompt. Sentences that had a task push or record now name a developer working that task. The #728 paragraph says export and import share PromptClient calls instead of promising prompt get --all shows an export file's fields. The assistant spec's removal requirement heading names import and the Agent Server. #807 Export, import, compare, and verify prompts and assistants across LangSmith environments Co-Authored-By: Claude Opus 5.5 --- .../design.md | 16 ++++++++-------- .../proposal.md | 2 +- .../specs/assistant/spec.md | 2 +- 3 files changed, 10 insertions(+), 10 deletions(-) diff --git a/openspec/changes/export-import-prompts-and-assistants/design.md b/openspec/changes/export-import-prompts-and-assistants/design.md index 511fb5aff..6367b9f91 100644 --- a/openspec/changes/export-import-prompts-and-assistants/design.md +++ b/openspec/changes/export-import-prompts-and-assistants/design.md @@ -55,7 +55,7 @@ Import, compare and verify read a file only when its `format_version` is 1 and i ### Export asks LangSmith for the model -Export reads the commit with `include=model`, the form the Go SDK uses (`references/langsmith-go.md`). The Python SDK sends the older `include_model=true`, and the LangSmith OpenAPI spec accepts both. With the model included, the manifest wraps the prompt and the chat model with its settings in one constructor, and the response also carries `model_config` and `model_provider` beside the manifest (`references/langsmith-sdk.md`). No field of the create-commit request holds the model, so the model reaches the destination inside the manifest. [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8) pushes that manifest to a second workspace, records whether LangSmith returns the same model, and records which outer `id` the live API returns. +Export reads the commit with `include=model`, the form the Go SDK uses (`references/langsmith-go.md`). The Python SDK sends the older `include_model=true`, and the LangSmith OpenAPI spec accepts both. With the model included, the manifest wraps the prompt and the chat model with its settings in one constructor, and the response also carries `model_config` and `model_provider` beside the manifest (`references/langsmith-sdk.md`). No field of the create-commit request holds the model, so the model reaches the destination inside the manifest. In [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8), a developer pushes that manifest to a second workspace and records whether LangSmith returns the same model, and which outer `id` the live API returns. ### Import records what it created in a separate file @@ -65,7 +65,7 @@ Import writes the destination's IDs to `.destinations.json`, keyed by A prompt that exists in the destination gets a new commit only with `--update`, and its parent is the destination's latest commit, so LangSmith keeps the earlier history. Import sends the file's commit description with each commit it creates. With `--update`, import also sets the destination repo's description, readme, tags and visibility to the file's values through `PATCH /api/v1/repos/{owner}/{repo}` when they differ. When only those repo fields differ and the manifest matches, import updates the repo and adds no commit. A commit tag in the file that the destination lacks, or that points at another commit there, also counts as a difference; with `--update`, import creates or moves the tag to the destination commit that matches the file, and adds no commit when the latest commit already matches. The tag endpoints take a commit's `id`, which differs from its hash. The commit read and the repo read return no `id`, so import looks the commit up in the commit list, `GET /api/v1/commits/{owner}/{repo}`, whose entries carry both the `id` and the hash. -With `--update`, import makes an existing assistant match the file and asks the Agent Server to remove what the file lacks. The Agent Server keeps the name, graph ID and description an update leaves out, and merges the metadata an update sends into the metadata the assistant already has, as `AssistantPatch` in `reference/openapi/langchain/agent-server/openapi.json` documents. So import sends the file's name, graph ID, configuration and context in full, an empty object for a configuration or context the file lacks, and a null value for each metadata key the destination has and the file lacks. `AssistantPatch` documents no way to clear a description, so [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8) tries a null and an empty description against the live server, along with a null metadata value and a replaced configuration and context. Import then reads the assistant back, and any key or field the file lacks that is still there shows up as a difference, with exit status 3. The Agent Server saves each update as a new version of the assistant. +With `--update`, import makes an existing assistant match the file and asks the Agent Server to remove what the file lacks. The Agent Server keeps the name, graph ID and description an update leaves out, and merges the metadata an update sends into the metadata the assistant already has, as `AssistantPatch` in `reference/openapi/langchain/agent-server/openapi.json` documents. So import sends the file's name, graph ID, configuration and context in full, an empty object for a configuration or context the file lacks, and a null value for each metadata key the destination has and the file lacks. `AssistantPatch` documents no way to clear a description, so in [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8) a developer sends a null and an empty description to the live server, along with a null metadata value and a replaced configuration and context, and records what the server keeps. Import then reads the assistant back, and any key or field the file lacks that is still there shows up as a difference, with exit status 3. The Agent Server saves each update as a new version of the assistant. Without `--update`, import lists the differences and exits with status 3. `--dry-run` reports the action import would take and writes nothing. @@ -77,7 +77,7 @@ Compare reads the destination, removes the fields the server sets (IDs, version Prompt verify checks that the workspace holds every secret the prompt's model reads. Export lists those names in `requires.secrets` by walking the manifest for LangChain secret references, `{"lc": 1, "type": "secret", "id": ["OPENAI_API_KEY"]}`, the shape the Python SDK's `_manifest_has_secrets` matches. Verify looks each name up among the workspace's secrets (`GET /api/v1/workspaces/current/secrets`, which `langstar secrets list` already calls) and prints the name of each missing secret and no secret value. -Prompt verify also notes when the workspace has no saved model configuration for the prompt's model, and leaves the exit status alone, because the prompt runs with the model its manifest carries. Export takes the model's class name, such as `ChatOpenAI`, from the last element of the chat model constructor's `id` in the manifest, such as `["langchain", "chat_models", "openai", "ChatOpenAI"]`, and stores it in `requires.model_provider`. Verify looks for a saved model configuration (`GET /api/v1/playground-settings`, which `langstar model-config list` calls) whose settings `id` ends in that class name. LangSmith also returns a `model_provider` field beside the manifest, which the file keeps as `commit.model_provider`; no SDK or spec shows what it holds, so [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8) records it. [#807 Export, import, compare, and verify prompts and assistants across LangSmith environments](https://github.com/codekiln/langstar/issues/807) asks for the note, and [Should prompt verify report a missing saved model configuration?](#7---should-prompt-verify-report-a-missing-saved-model-configuration) asks whether to keep it. +Prompt verify also notes when the workspace has no saved model configuration for the prompt's model, and leaves the exit status alone, because the prompt runs with the model its manifest carries. Export takes the model's class name, such as `ChatOpenAI`, from the last element of the chat model constructor's `id` in the manifest, such as `["langchain", "chat_models", "openai", "ChatOpenAI"]`, and stores it in `requires.model_provider`. Verify looks for a saved model configuration (`GET /api/v1/playground-settings`, which `langstar model-config list` calls) whose settings `id` ends in that class name. LangSmith also returns a `model_provider` field beside the manifest, which the file keeps as `commit.model_provider`; no SDK or spec shows what it holds, so in [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8) a developer records the value the live API returns. [#807 Export, import, compare, and verify prompts and assistants across LangSmith environments](https://github.com/codekiln/langstar/issues/807) asks for the note, and [Should prompt verify report a missing saved model configuration?](#7---should-prompt-verify-report-a-missing-saved-model-configuration) asks whether to keep it. Assistant verify checks the graph with `POST /assistants/search` filtered to the graph ID; every graph on a deployment has a default assistant, so an empty result means the deployment does not serve the graph. It checks each prompt in `requires.prompts`, such as `support-system:prod`, by asking LangSmith for the commit that tag points at (`GET /api/v1/commits/{owner}/{repo}/{tag}`, the request `langstar prompt pull --commit prod` already sends). A 404 means the workspace lacks the prompt or its tag. @@ -95,17 +95,17 @@ Status 0 means the resource matches and every check passed. Status 3 means langs ### A Python script checks that an imported structured prompt still returns structured data -A mise task runs a short Python script that pulls the imported prompt with `include_model=True`, invokes it, and checks that the result matches the prompt's schema. The script passes `secrets={"OPENAI_API_KEY": }` to [`pull_prompt`](https://github.com/langchain-ai/langsmith-sdk/blob/47c553caf08c1989e2be3ff1886da3eb801f709f/python/langsmith/client.py#L10228), because the LangSmith Python SDK reads no secret from the environment unless asked and raises when a manifest holds a secret reference and no `secrets` map arrives. The script exists because the LangSmith API has no documented way to run a prompt with its model: the one prompt endpoint in the OpenAPI spec, `POST /api/v1/prompts/invoke_prompt`, takes message templates and inputs with no model or schema, and no LangSmith SDK calls it. [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8) tries that endpoint against the live API in case it does more than the spec says. The script needs a model provider API key, which codekiln has to create and add to fnox and CI before the test can run. +A mise task runs a short Python script that pulls the imported prompt with `include_model=True`, invokes it, and checks that the result matches the prompt's schema. The script passes `secrets={"OPENAI_API_KEY": }` to [`pull_prompt`](https://github.com/langchain-ai/langsmith-sdk/blob/47c553caf08c1989e2be3ff1886da3eb801f709f/python/langsmith/client.py#L10228), because the LangSmith Python SDK reads no secret from the environment unless asked and raises when a manifest holds a secret reference and no `secrets` map arrives. The script exists because the LangSmith API has no documented way to run a prompt with its model: the one prompt endpoint in the OpenAPI spec, `POST /api/v1/prompts/invoke_prompt`, takes message templates and inputs with no model or schema, and no LangSmith SDK calls it. In [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8), a developer calls that endpoint on the live API in case it does more than the spec says. The script needs a model provider API key, which codekiln has to create and add to fnox and CI before the test can run. ### Export and import use the same SDK calls as the planned `prompt create`, `get` and `update` -[#728 Implement CRUD command structure (create, get, update) for prompt commands](https://github.com/codekiln/langstar/issues/728) splits prompt text, model and structured output into separate `create`, `get` and `update` commands for editing one field at a time. Export and import handle the whole prompt at once. Both use the same SDK calls, and once #728 lands, `prompt get --all` shows the same fields an export file holds. +[#728 Implement CRUD command structure (create, get, update) for prompt commands](https://github.com/codekiln/langstar/issues/728) plans `prompt create`, `get` and `update` commands that edit a prompt's text, model or structured output one field at a time. Export and import read and write the whole prompt at once, through the same `PromptClient` calls in `sdk/src/prompts.rs` that those commands will use, so a field the SDK reads for export is there for `prompt get` too. ## Risks / Trade-offs -- [LangSmith may store a pushed model differently from the source] → Import reads the prompt back and reports any difference, and [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8) checks the round trip before the commands are built. +- [LangSmith may store a pushed model differently from the source] → Import reads the prompt back and reports any difference, and in [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8) a developer checks the round trip against the live API before anyone builds the commands. - [Pushing a manifest identical to the latest commit may return 409] → Import compares first and pushes nothing when the prompt matches. -- [langstar finds a deployment's Agent Server URL only in `source_config.custom_url` (`resolve_deployment_url` in `cli/src/deployment_utils.rs`), and a deployment built from GitHub has none, so every `langstar assistant` command, the new ones included, fails on such a deployment] → [export-import-prompts-and-assistants Task 2.2 Return the URL of a deployment built from GitHub](tasks.md#L13) adds the control plane's top-level `url` to the SDK's `Deployment`, which drops it today, and makes `resolve_deployment_url` use it before anyone builds the assistant export, compare, verify and import commands. +- [langstar finds a deployment's Agent Server URL only in `source_config.custom_url` (`resolve_deployment_url` in `cli/src/deployment_utils.rs`), and a deployment built from GitHub has none, so every `langstar assistant` command, the new ones included, fails on such a deployment] → In [export-import-prompts-and-assistants Task 2.2 Return the URL of a deployment built from GitHub](tasks.md#L13), a developer adds the control plane's top-level `url` to the SDK's `Deployment`, which drops it today, and makes `resolve_deployment_url` use it, before anyone builds the assistant export, compare, verify and import commands. - [A tag in the destination may already point at another commit] → `--update` moves the tag, and `--dry-run` lists the move. ## Resolved Questions @@ -131,7 +131,7 @@ None. ### 3 - How should the end-to-end test show that an imported structured prompt still returns structured data? > Context from the drafting agent, for question 3. -> The LangSmith OpenAPI spec's only prompt-invoke endpoint, `POST /api/v1/prompts/invoke_prompt`, takes message templates and inputs with no model or schema, and no SDK calls it. [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8) tries it live. The draft runs a Python script through a mise task, using the LangSmith Python SDK and LangChain, which is how most users run a prompt. It needs a model provider key, such as `OPENAI_API_KEY`, which codekiln would create and add to fnox and CI. Building the model call into langstar as `prompt verify --invoke` would need langstar to call each provider's API itself. Recommendation: the Python script, with an OpenAI key and a small model. +> The LangSmith OpenAPI spec's only prompt-invoke endpoint, `POST /api/v1/prompts/invoke_prompt`, takes message templates and inputs with no model or schema, and no SDK calls it. In [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8), a developer calls it on the live API. The draft runs a Python script through a mise task, using the LangSmith Python SDK and LangChain, which is how most users run a prompt. It needs a model provider key, such as `OPENAI_API_KEY`, which codekiln would create and add to fnox and CI. Building the model call into langstar as `prompt verify --invoke` would need langstar to call each provider's API itself. Recommendation: the Python script, with an OpenAI key and a small model. diff --git a/openspec/changes/export-import-prompts-and-assistants/proposal.md b/openspec/changes/export-import-prompts-and-assistants/proposal.md index 950b82a52..e0c6af64f 100644 --- a/openspec/changes/export-import-prompts-and-assistants/proposal.md +++ b/openspec/changes/export-import-prompts-and-assistants/proposal.md @@ -14,7 +14,7 @@ A prompt belongs to a workspace and an assistant belongs to a deployment, but bo - `langstar prompt import --workspace-id ` creates the prompt in the destination workspace, or adds a commit to an existing prompt, updates its repo details and moves its commit tags when the user passes `--update`. `--dry-run` shows what it would do. After writing, it reads the prompt back and reports any difference. - `langstar prompt compare ` reports the differences between the file and a prompt in a workspace. `langstar prompt verify ` also checks that the workspace has the secrets the prompt's model reads, and notes when the workspace has no saved model configuration for the model's provider. - `langstar assistant export`, `import`, `compare` and `verify` do the same for assistants on a deployment. The file holds the name, description, graph ID, configuration, context, metadata and any prompt names the assistant refers to. `verify` checks that the destination deployment has the graph and that the destination workspace has the referenced prompts. -- Import records each destination ID (the new commit hash, or the new assistant ID) in a separate file next to the export, so importing the same file again finds the copy it made earlier instead of creating another one. +- Import records each destination ID (the new commit hash, or the new assistant ID) in a separate file next to the export. When a user imports the same file again, langstar reads that file to find the assistant it created earlier instead of creating another one, and to tell whether someone pushed a commit to the prompt since the last import. - Import runs the dependency checks of `verify` before it writes, and writes nothing when one fails. - `compare`, `verify` and `import` share one exit status scheme, 0 for a match, 3 for a difference or a failed check and 1 for any other error, and print a JSON report with `-f json`. Status 2 stays clap's usage error. diff --git a/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md b/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md index 295d01b6f..52183cee7 100644 --- a/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md +++ b/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md @@ -109,7 +109,7 @@ When the destination record holds an assistant ID for the deployment, `langstar - **WHEN** the user runs `langstar assistant import support-bot.json --deployment my-agent-prod --update --assistant-id ` - **THEN** langstar updates that assistant and records its ID for `my-agent-prod` -### Requirement: Remove on update what the file lacks +### Requirement: Import with --update asks the Agent Server to remove the metadata, description and context the file lacks `langstar assistant import --update` SHALL ask the Agent Server to remove each metadata key the destination assistant has and the file lacks, and to clear a description or context the file lacks, then read the assistant back. When such a key or field remains after the update, it SHALL list it as a difference and exit with status 3. From 7c3832b0a023f4fdcae67d34b24a76e5cc2ae648 Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Tue, 13 Oct 2026 20:24:19 -0400 Subject: [PATCH 20/90] =?UTF-8?q?=F0=9F=93=9A=20docs(openspec):=20mark=20t?= =?UTF-8?q?he=20SDK=20field=20additions=20as=20breaking,=20compare=20every?= =?UTF-8?q?=20manifest=20key,=20and=20check=20prompt=20references=20agains?= =?UTF-8?q?t=20the=20file?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adding a public field to Deployment, CommitData, CommitRequest, Assistant or the assistant request structs breaks a struct literal in another crate, so those tasks update the literals in this repository and their pull requests carry a BREAKING CHANGE footer. Compare drops no key by name inside the repo, commit and assistant objects, so a manifest's id arrays count. Assistant compare, verify and import resolve each requires.prompts pointer against the file's assistant object and stop with status 1 on a mismatch. #807 Export, import, compare, and verify prompts and assistants across LangSmith environments Co-Authored-By: Claude Opus 5.5 --- .../export-import-prompts-and-assistants/design.md | 5 +++-- .../proposal.md | 2 +- .../specs/assistant/spec.md | 13 ++++++++++++- .../specs/prompt/spec.md | 7 ++++++- .../export-import-prompts-and-assistants/tasks.md | 8 +++++--- 5 files changed, 27 insertions(+), 8 deletions(-) diff --git a/openspec/changes/export-import-prompts-and-assistants/design.md b/openspec/changes/export-import-prompts-and-assistants/design.md index 6367b9f91..6381db6c9 100644 --- a/openspec/changes/export-import-prompts-and-assistants/design.md +++ b/openspec/changes/export-import-prompts-and-assistants/design.md @@ -71,7 +71,7 @@ Without `--update`, import lists the differences and exits with status 3. `--dry ### Compare reports differences by JSON path -Compare reads the destination, removes the fields the server sets (IDs, version numbers, timestamps, owner and like counts), and walks both JSON documents. Compare looks only at the `repo`, `commit` and `assistant` objects. The source environment's workspace, deployment and IDs sit in `source`, so they never show up as differences against a copy. Each difference names its JSON Pointer path and both values. For a prompt, the comparison covers the commit description, the commit tags and the commit each one points at, and the repo's description, readme, tags and visibility as well as the manifest. Import runs the same comparison after it writes, so a field LangSmith or the Agent Server drops shows up as a difference. +Compare reads the destination into the export file's `repo`, `commit` and `assistant` objects, which hold none of the fields the server sets (IDs, version numbers, timestamps, owner and like counts), and walks both JSON documents. Compare drops no key by name inside those objects, so the `id` arrays inside a manifest, which name the prompt and model classes such as `ChatOpenAI`, count as differences. The source environment's workspace, deployment and IDs sit in `source`, so they never show up as differences against a copy. Each difference names its JSON Pointer path and both values. For a prompt, the comparison covers the commit description, the commit tags and the commit each one points at, and the repo's description, readme, tags and visibility as well as the manifest. Import runs the same comparison after it writes, so a field LangSmith or the Agent Server drops shows up as a difference. ### Verify checks what the copy needs to run @@ -87,7 +87,7 @@ Before it writes anything, import checks that the destination has the secrets, t ### The user names an assistant's prompt references -An assistant refers to a prompt through whatever configuration key its graph reads, so langstar cannot find the references by itself. `assistant export --prompt-ref ` names each one. The assistant's configuration should name each prompt by handle and tag, such as `support-system:prod`, because the same prompt gets a different commit hash in each workspace. Import copies the commit tags to the destination workspace, so when the graph on the destination deployment asks LangSmith for `support-system:prod`, it gets the same prompt. +An assistant refers to a prompt through whatever configuration key its graph reads, so langstar cannot find the references by itself. `assistant export --prompt-ref ` names each one. The assistant's configuration should name each prompt by handle and tag, such as `support-system:prod`, because the same prompt gets a different commit hash in each workspace. Import copies the commit tags to the destination workspace, so when the graph on the destination deployment asks LangSmith for `support-system:prod`, it gets the same prompt. Because `requires.prompts` repeats a value the configuration already holds, someone can edit one in Git and not the other, so compare, verify and import resolve each pointer against the file's `assistant` object when they read the file, and stop with status 1 when the value there differs from the recorded prompt name. ### compare, verify and import exit with 3 when they find a difference @@ -106,6 +106,7 @@ A mise task runs a short Python script that pulls the imported prompt with `incl - [LangSmith may store a pushed model differently from the source] → Import reads the prompt back and reports any difference, and in [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8) a developer checks the round trip against the live API before anyone builds the commands. - [Pushing a manifest identical to the latest commit may return 409] → Import compares first and pushes nothing when the prompt matches. - [langstar finds a deployment's Agent Server URL only in `source_config.custom_url` (`resolve_deployment_url` in `cli/src/deployment_utils.rs`), and a deployment built from GitHub has none, so every `langstar assistant` command, the new ones included, fails on such a deployment] → In [export-import-prompts-and-assistants Task 2.2 Return the URL of a deployment built from GitHub](tasks.md#L13), a developer adds the control plane's top-level `url` to the SDK's `Deployment`, which drops it today, and makes `resolve_deployment_url` use it, before anyone builds the assistant export, compare, verify and import commands. +- [Tasks 2.2 and 3.1 to 3.4 add public fields to SDK structs, so a crate that builds one of those structs with a struct literal stops compiling] → Each of those pull requests updates every struct literal of the changed struct in this repository and carries a `BREAKING CHANGE:` footer, which `docs/dev/git-scm-conventions.md` says triggers a major version bump. - [A tag in the destination may already point at another commit] → `--update` moves the tag, and `--dry-run` lists the move. ## Resolved Questions diff --git a/openspec/changes/export-import-prompts-and-assistants/proposal.md b/openspec/changes/export-import-prompts-and-assistants/proposal.md index e0c6af64f..beed518d5 100644 --- a/openspec/changes/export-import-prompts-and-assistants/proposal.md +++ b/openspec/changes/export-import-prompts-and-assistants/proposal.md @@ -30,7 +30,7 @@ A prompt belongs to a workspace and an assistant belongs to a deployment, but bo ## Impact -- SDK: `sdk/src/prompts.rs` gains a commit read that asks for the model, a commit description on commit create, the new commit's `id`, a repo update, a commit list that returns each commit's `id`, and calls that list, create and move commit tags. `sdk/src/assistants.rs` gains the `context` and `version` fields, `graph_id` on update, and a search by graph ID. `sdk/src/deployments.rs` gains the deployment's `url`. +- SDK: `sdk/src/prompts.rs` gains a commit read that asks for the model, a commit description on commit create, the new commit's `id`, a repo update, a commit list that returns each commit's `id`, and calls that list, create and move commit tags. `sdk/src/assistants.rs` gains the `context` and `version` fields, `graph_id` on update, and a search by graph ID. `sdk/src/deployments.rs` gains the deployment's `url`. Each new public field breaks a crate that builds that SDK struct with a struct literal, so the pull requests that add them carry a `BREAKING CHANGE:` footer, and the release that ships them is a major version. - CLI: new subcommands in `cli/src/commands/prompt.rs` and `cli/src/commands/assistant.rs`, and a shared module for the export file, the destination record, comparison and the exit status. `resolve_deployment_url` in `cli/src/deployment_utils.rs` also returns the Agent Server URL of a deployment built from GitHub, which every `langstar assistant` command needs. - Tests need a second workspace, a second deployment and one model provider key for the structured output check. codekiln creates the workspace and the key, and langstar's tests read them through `fnox.toml` and the CI workflow; the `TestDeploymentConfig` fixture creates the deployment on the first run and the tests reuse it after that. - Docs: a new usage page on moving prompts and assistants between environments. diff --git a/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md b/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md index 52183cee7..d62b4df70 100644 --- a/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md +++ b/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md @@ -154,7 +154,7 @@ After writing, `langstar assistant import` SHALL record the assistant ID in the ### Requirement: Compare an assistant with its export file -`langstar assistant compare --deployment ` SHALL compare the file with the recorded assistant, or the one named by `--assistant-id`, and list each difference by its JSON path with both values. Fields the server sets, such as the assistant ID, version and timestamps, SHALL be left out of the comparison. +`langstar assistant compare --deployment ` SHALL compare the file with the recorded assistant, or the one named by `--assistant-id`, and list each difference by its JSON path with both values. It SHALL compare only the fields the export file holds, leaving out the assistant ID, version and timestamps the server sets. #### Scenario: Metadata changed on the deployment @@ -186,6 +186,17 @@ After writing, `langstar assistant import` SHALL record the assistant ID in the - **AND** `my-prompt.json` holds `"kind": "prompt"` - **THEN** langstar creates no assistant, exits with status 1 and names `kind` and the value `prompt` +### Requirement: Reject a file whose prompt reference no longer matches its assistant + +`assistant import`, `assistant compare` and `assistant verify` SHALL resolve each pointer under `requires.prompts` against the file's `assistant` object when they read the file. When a pointer resolves to no string, or to a string other than the prompt name recorded with it, they SHALL write nothing, exit with status 1 and name the pointer and both values. + +#### Scenario: Configuration edited after export + +- **WHEN** `requires.prompts` records `/config/configurable/system_prompt` with `support-system:prod` +- **AND** the file's `assistant.config.configurable.system_prompt` is `support-system:staging` +- **AND** the user runs `langstar assistant verify support-bot.json --deployment my-agent-prod` +- **THEN** langstar exits with status 1 and names the pointer, `support-system:prod` and `support-system:staging` + ### Requirement: Assistant compare, verify and import use the prompt commands' exit statuses and JSON report `assistant compare`, `assistant verify` and `assistant import` SHALL use the exit statuses and the JSON report shape of `prompt compare`, `prompt verify` and `prompt import`. diff --git a/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md index 324ed520f..bbcaf3c23 100644 --- a/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md +++ b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md @@ -172,7 +172,7 @@ After writing, `langstar prompt import` SHALL record the commit hash in the dest ### Requirement: Compare a prompt with its export file -`langstar prompt compare ` SHALL read the prompt with the file's handle from the workspace, with the model included, and its repo details, and list each difference by its JSON path with the file's value and the workspace's value. `--commit ` SHALL choose the commit; without it langstar SHALL read the latest commit. Fields the server sets, such as IDs and timestamps, SHALL be left out of the comparison. +`langstar prompt compare ` SHALL read the prompt with the file's handle from the workspace, with the model included, and its repo details, and list each difference by its JSON path with the file's value and the workspace's value. `--commit ` SHALL choose the commit; without it langstar SHALL read the latest commit. It SHALL compare only the fields the file holds, leaving out server-set IDs and timestamps, and every key in the manifest, its `id` arrays included. #### Scenario: Prompt matches @@ -184,6 +184,11 @@ After writing, `langstar prompt import` SHALL record the commit hash in the dest - **WHEN** someone changed the model temperature in the workspace's prompt - **THEN** langstar lists the temperature's JSON path with both values and exits with status 3 +#### Scenario: Model class changed + +- **WHEN** someone changed the workspace's prompt to use `ChatAnthropic` where the file's manifest names `ChatOpenAI` +- **THEN** langstar lists the JSON path of the model's `id` array with both values and exits with status 3 + #### Scenario: Repo readme changed - **WHEN** someone changed the readme of the workspace's prompt diff --git a/openspec/changes/export-import-prompts-and-assistants/tasks.md b/openspec/changes/export-import-prompts-and-assistants/tasks.md index 9d25fc7cb..4b48efe5a 100644 --- a/openspec/changes/export-import-prompts-and-assistants/tasks.md +++ b/openspec/changes/export-import-prompts-and-assistants/tasks.md @@ -10,11 +10,13 @@ Each numbered group below is one pull request, merged in the order listed. Task ## 2. Test environments - [ ] 2.1 Add the destination workspace and the model provider key to `fnox.toml`, the CI workflow and `docs/dev/environment-variables.md`, with placeholders only; verify with `mise run check` and a CI run that reads both. This task waits on codekiln to create a second workspace in the test organization and an OpenAI API key, and to store both in 1Password and the repository's CI secrets and the key as the destination workspace's `OPENAI_API_KEY` secret. The import tests need the workspace as a place to import into, and its secret lets prompt import's secret check pass for a prompt with an OpenAI model. The key lets the check in 7.3 run each imported structured prompt once against a small OpenAI model, which bills that account on every CI run; it is the one test that shows a copy still returns structured data, where every other test compares text. -- [ ] 2.2 Add `url` to `Deployment` in `sdk/src/deployments.rs`, with a method that returns `url` and falls back to `source_config.custom_url`, and make `resolve_deployment_url` in `cli/src/deployment_utils.rs` call that method, so that a deployment built from GitHub resolves; add a unit test for each case in `sdk/src/deployments.rs`, and verify by removing `#[ignore]` from `test_assistant_create_basic` in `cli/tests/assistant_command_test.rs` and running it against the test deployment +- [ ] 2.2 Add `url` to `Deployment` in `sdk/src/deployments.rs`, with a method that returns `url` and falls back to `source_config.custom_url`, and make `resolve_deployment_url` in `cli/src/deployment_utils.rs` call that method, so that a deployment built from GitHub resolves; end the four `Deployment` struct literals in `sdk/tests/integration_deployment_workflow.rs` and `cli/src/commands/deployment.rs` with `..Default::default()`, and give the pull request a `BREAKING CHANGE:` footer, because a crate that builds a `Deployment` with a struct literal stops compiling when the field is added; add a unit test for each case in `sdk/src/deployments.rs`, and verify by removing `#[ignore]` from `test_assistant_create_basic` in `cli/tests/assistant_command_test.rs` and running it against the test deployment - [ ] 2.3 Add a destination deployment fixture to `sdk/src/test_utils.rs` and `cli/tests/common/fixtures.rs` that builds the test graph from its own `langgraph.json` under `tests/fixtures/`, because the control plane allows one deployment per repository and graph config file and `find_reusable_deployment` would otherwise hand back the source deployment; verify with an integration test that asserts the destination and source deployment IDs differ and lists assistants on the destination deployment through the URL that 2.2 resolves ## 3. SDK reads and writes the fields export and import need +Each task in this group adds a public field to an SDK struct (`CommitData`, `CommitRequest`, `Assistant` and the assistant request structs), which breaks a crate that builds that struct with a struct literal. Each task updates every struct literal of the struct it changes in this repository, and its pull request carries a `BREAKING CHANGE:` footer. + - [ ] 3.1 Add a commit read that sends `include=model` and returns `description`, `model_config` and `model_provider` to `sdk/src/prompts.rs`; verify with an httpmock test that asserts the query string and the parsed fields, and an integration test that reads a prompt with a model - [ ] 3.2 Add the commit's `id` to `CommitData` in `sdk/src/prompts.rs`, a call that lists a prompt's commits with their `id` and hash (`GET /api/v1/commits/{owner}/{repo}`), and calls that list, create and move commit tags, which name their commit by that `id` in `POST /api/v1/repos/{owner}/{repo}/tags` and `PATCH /api/v1/repos/{owner}/{repo}/tags/{tag_name}`; verify with httpmock tests that assert `commit_id` in each request body, and an integration test that pushes two commits, tags the first, finds the second commit's `id` through the commit list, moves the tag to it, and reads the tag back through the SDK - [ ] 3.3 Add `context` and `version` to `Assistant`, `context` and `graph_id` to `UpdateAssistantRequest`, `context` to `CreateAssistantRequest`, and a `graph_id` filter to `AssistantSearchRequest`, in `sdk/src/assistants.rs`; verify with httpmock tests of the request bodies and an integration test that creates an assistant with context, reads it back with its version, finds it by graph ID, and updates its graph ID and reads the new graph ID back @@ -23,7 +25,7 @@ Each numbered group below is one pull request, merged in the order listed. Task ## 4. Shared code for the prompt and assistant commands: the export file, the comparison and the report - [ ] 4.1 Add the export file types, with sorted keys, no export time and a reader that accepts only `format_version` 1 and the expected `kind`, and the destination record file, to a new module in `cli/src/`; verify with unit tests that build equivalent nested values with keys inserted in different orders, serialize each, and assert identical text with keys sorted at every level, and that updating one destination leaves the others unchanged, and that reading a file whose `format_version` is 2, or whose `kind` names the other resource, returns an error naming the field and its value -- [ ] 4.2 Add the JSON comparison that drops server-set fields and reports differences by JSON Pointer; verify with unit tests on nested objects, arrays, missing keys and ignored fields +- [ ] 4.2 Add the JSON comparison, which compares every key under the `repo`, `commit` and `assistant` objects and reports differences by JSON Pointer; verify with unit tests on nested objects, arrays and missing keys, and a test that a changed `id` array inside `commit.manifest`, such as `ChatOpenAI` replaced by `ChatAnthropic`, is reported at its path - [ ] 4.3 Add the shared result report and the exit statuses 0, 3 and 1, printing JSON to stdout and progress to stderr, and leave clap's status 2 for usage errors; verify with unit tests of the report and of the exit status each result gives - [ ] 4.4 Add the secret reference walk that lists secret names from a manifest; verify with unit tests on a copy of the `prompt_with_model` manifest from the LangSmith Python SDK's integration tests, which `references/langsmith-sdk.md` links, and on a manifest with no secret reference @@ -52,7 +54,7 @@ Each numbered group below is one pull request, merged in the order listed. Task ## 9. assistant compare and verify -- [ ] 9.1 Add `langstar assistant compare` with `--assistant-id`, and the lookup of the recorded assistant in the destination record from 4.1; verify with an integration test that exports an assistant, changes its metadata through the SDK, runs compare with `--assistant-id` set to that assistant, and asserts the reported path and exit status 3, and a unit test that compare reads the assistant ID from a destination record written by the 4.1 types +- [ ] 9.1 Add `langstar assistant compare` with `--assistant-id`, and the lookup of the recorded assistant in the destination record from 4.1; verify with an integration test that exports an assistant, changes its metadata through the SDK, runs compare with `--assistant-id` set to that assistant, and asserts the reported path and exit status 3, and a unit test that compare reads the assistant ID from a destination record written by the 4.1 types; add the check, shared by assistant compare, verify and import, that each pointer under `requires.prompts` still resolves in the file's `assistant` object to the prompt name recorded with it, and verify with a unit test that a file whose configuration names another prompt at that pointer exits with status 1 and names the pointer and both values - [ ] 9.2 Add `langstar assistant verify` with `--assistant-id`, `--workspace-id` for the workspace that holds the prompts, the graph check, and the prompt check, which reads `GET /api/v1/commits/{owner}/{repo}/{tag}`; verify with integration tests that run verify with `--assistant-id` against an exported assistant, one with a file whose graph ID the deployment lacks, one with a missing prompt tag, and one that passes `--workspace-id` set to the destination workspace, and a CLI test that `langstar -f json assistant verify` prints one JSON object with `status`, `differences` and `checks` to stdout - [ ] 9.3 Document assistant compare and verify; verify that each documented command runs as written From f47fc4f28092ed5ca0fc552f8a461b0f87250c5c Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Tue, 13 Oct 2026 20:29:46 -0400 Subject: [PATCH 21/90] =?UTF-8?q?=F0=9F=93=9A=20docs(openspec):=20ask=20co?= =?UTF-8?q?dekiln=20whether=20the=20SDK=20field=20additions=20are=20a=20br?= =?UTF-8?q?eaking=20change?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add Open Question 9 with the recommendation to ship the SDK field additions as feat changes, and point tasks.md, the design's risk entry and the proposal at it. #807 Export, import, compare, and verify prompts and assistants across LangSmith environments Co-Authored-By: Claude Opus 5.5 --- .../export-import-prompts-and-assistants/design.md | 9 ++++++++- .../export-import-prompts-and-assistants/proposal.md | 2 +- .../export-import-prompts-and-assistants/tasks.md | 4 ++-- 3 files changed, 11 insertions(+), 4 deletions(-) diff --git a/openspec/changes/export-import-prompts-and-assistants/design.md b/openspec/changes/export-import-prompts-and-assistants/design.md index 6381db6c9..2f14ebb80 100644 --- a/openspec/changes/export-import-prompts-and-assistants/design.md +++ b/openspec/changes/export-import-prompts-and-assistants/design.md @@ -106,7 +106,7 @@ A mise task runs a short Python script that pulls the imported prompt with `incl - [LangSmith may store a pushed model differently from the source] → Import reads the prompt back and reports any difference, and in [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8) a developer checks the round trip against the live API before anyone builds the commands. - [Pushing a manifest identical to the latest commit may return 409] → Import compares first and pushes nothing when the prompt matches. - [langstar finds a deployment's Agent Server URL only in `source_config.custom_url` (`resolve_deployment_url` in `cli/src/deployment_utils.rs`), and a deployment built from GitHub has none, so every `langstar assistant` command, the new ones included, fails on such a deployment] → In [export-import-prompts-and-assistants Task 2.2 Return the URL of a deployment built from GitHub](tasks.md#L13), a developer adds the control plane's top-level `url` to the SDK's `Deployment`, which drops it today, and makes `resolve_deployment_url` use it, before anyone builds the assistant export, compare, verify and import commands. -- [Tasks 2.2 and 3.1 to 3.4 add public fields to SDK structs, so a crate that builds one of those structs with a struct literal stops compiling] → Each of those pull requests updates every struct literal of the changed struct in this repository and carries a `BREAKING CHANGE:` footer, which `docs/dev/git-scm-conventions.md` says triggers a major version bump. +- [Tasks 2.2 and 3.1 to 3.4 add public fields to SDK structs, so a crate that builds one of those structs with a struct literal stops compiling] → Each of those pull requests updates every struct literal of the changed struct in this repository. [Open Question 9](#9---should-the-sdk-field-additions-count-as-a-breaking-change) asks whether they also carry a `BREAKING CHANGE:` footer, which `docs/dev/git-scm-conventions.md` says triggers a major version bump. - [A tag in the destination may already point at another commit] → `--update` moves the tag, and `--dry-run` lists the move. ## Resolved Questions @@ -170,3 +170,10 @@ None. > [#807 Export, import, compare, and verify prompts and assistants across LangSmith environments](https://github.com/codekiln/langstar/issues/807) asks import to check the destination's dependencies: the secrets a prompt's model reads, and an assistant's graph and referenced prompts. The draft runs those checks before import writes and writes nothing when one fails, so a user never has to find and delete a copy that cannot run. The cost is order: the destination workspace needs its secrets before a prompt import, and its prompts before an assistant import. The other choice writes the copy anyway and reports the failed checks with status 3 afterwards, which lets a user import in any order and fix the destination later. Recommendation: stop before writing. + +### 9 - Should the SDK field additions count as a breaking change? + +> Context from the drafting agent, for question 9. +> Tasks 2.2 and 3.1 to 3.4 add public fields to `Deployment`, `CommitData`, `CommitRequest`, `Assistant` and the assistant request structs. A crate that builds one of those structs with a struct literal stops compiling, so by `docs/dev/git-scm-conventions.md` each of those pull requests would carry a `BREAKING CHANGE:` footer and the release would become a major version. `sdk/Cargo.toml` does not set `publish = false`, but the CLI in this workspace is the only user of the SDK that langstar knows of, and each pull request updates the struct literals in this repository. Recommendation: treat the additions as `feat` changes with no `BREAKING CHANGE:` footer, and name the new fields in the release notes. + + diff --git a/openspec/changes/export-import-prompts-and-assistants/proposal.md b/openspec/changes/export-import-prompts-and-assistants/proposal.md index beed518d5..510157af8 100644 --- a/openspec/changes/export-import-prompts-and-assistants/proposal.md +++ b/openspec/changes/export-import-prompts-and-assistants/proposal.md @@ -30,7 +30,7 @@ A prompt belongs to a workspace and an assistant belongs to a deployment, but bo ## Impact -- SDK: `sdk/src/prompts.rs` gains a commit read that asks for the model, a commit description on commit create, the new commit's `id`, a repo update, a commit list that returns each commit's `id`, and calls that list, create and move commit tags. `sdk/src/assistants.rs` gains the `context` and `version` fields, `graph_id` on update, and a search by graph ID. `sdk/src/deployments.rs` gains the deployment's `url`. Each new public field breaks a crate that builds that SDK struct with a struct literal, so the pull requests that add them carry a `BREAKING CHANGE:` footer, and the release that ships them is a major version. +- SDK: `sdk/src/prompts.rs` gains a commit read that asks for the model, a commit description on commit create, the new commit's `id`, a repo update, a commit list that returns each commit's `id`, and calls that list, create and move commit tags. `sdk/src/assistants.rs` gains the `context` and `version` fields, `graph_id` on update, and a search by graph ID. `sdk/src/deployments.rs` gains the deployment's `url`. Each new public field breaks a crate outside this repository that builds that SDK struct with a struct literal; design.md asks codekiln whether that makes the release a major version. - CLI: new subcommands in `cli/src/commands/prompt.rs` and `cli/src/commands/assistant.rs`, and a shared module for the export file, the destination record, comparison and the exit status. `resolve_deployment_url` in `cli/src/deployment_utils.rs` also returns the Agent Server URL of a deployment built from GitHub, which every `langstar assistant` command needs. - Tests need a second workspace, a second deployment and one model provider key for the structured output check. codekiln creates the workspace and the key, and langstar's tests read them through `fnox.toml` and the CI workflow; the `TestDeploymentConfig` fixture creates the deployment on the first run and the tests reuse it after that. - Docs: a new usage page on moving prompts and assistants between environments. diff --git a/openspec/changes/export-import-prompts-and-assistants/tasks.md b/openspec/changes/export-import-prompts-and-assistants/tasks.md index 4b48efe5a..31dde9124 100644 --- a/openspec/changes/export-import-prompts-and-assistants/tasks.md +++ b/openspec/changes/export-import-prompts-and-assistants/tasks.md @@ -10,12 +10,12 @@ Each numbered group below is one pull request, merged in the order listed. Task ## 2. Test environments - [ ] 2.1 Add the destination workspace and the model provider key to `fnox.toml`, the CI workflow and `docs/dev/environment-variables.md`, with placeholders only; verify with `mise run check` and a CI run that reads both. This task waits on codekiln to create a second workspace in the test organization and an OpenAI API key, and to store both in 1Password and the repository's CI secrets and the key as the destination workspace's `OPENAI_API_KEY` secret. The import tests need the workspace as a place to import into, and its secret lets prompt import's secret check pass for a prompt with an OpenAI model. The key lets the check in 7.3 run each imported structured prompt once against a small OpenAI model, which bills that account on every CI run; it is the one test that shows a copy still returns structured data, where every other test compares text. -- [ ] 2.2 Add `url` to `Deployment` in `sdk/src/deployments.rs`, with a method that returns `url` and falls back to `source_config.custom_url`, and make `resolve_deployment_url` in `cli/src/deployment_utils.rs` call that method, so that a deployment built from GitHub resolves; end the four `Deployment` struct literals in `sdk/tests/integration_deployment_workflow.rs` and `cli/src/commands/deployment.rs` with `..Default::default()`, and give the pull request a `BREAKING CHANGE:` footer, because a crate that builds a `Deployment` with a struct literal stops compiling when the field is added; add a unit test for each case in `sdk/src/deployments.rs`, and verify by removing `#[ignore]` from `test_assistant_create_basic` in `cli/tests/assistant_command_test.rs` and running it against the test deployment +- [ ] 2.2 Add `url` to `Deployment` in `sdk/src/deployments.rs`, with a method that returns `url` and falls back to `source_config.custom_url`, and make `resolve_deployment_url` in `cli/src/deployment_utils.rs` call that method, so that a deployment built from GitHub resolves; end the four `Deployment` struct literals in `sdk/tests/integration_deployment_workflow.rs` and `cli/src/commands/deployment.rs` with `..Default::default()`, because a crate that builds a `Deployment` with a struct literal stops compiling when the field is added; add a unit test for each case in `sdk/src/deployments.rs`, and verify by removing `#[ignore]` from `test_assistant_create_basic` in `cli/tests/assistant_command_test.rs` and running it against the test deployment - [ ] 2.3 Add a destination deployment fixture to `sdk/src/test_utils.rs` and `cli/tests/common/fixtures.rs` that builds the test graph from its own `langgraph.json` under `tests/fixtures/`, because the control plane allows one deployment per repository and graph config file and `find_reusable_deployment` would otherwise hand back the source deployment; verify with an integration test that asserts the destination and source deployment IDs differ and lists assistants on the destination deployment through the URL that 2.2 resolves ## 3. SDK reads and writes the fields export and import need -Each task in this group adds a public field to an SDK struct (`CommitData`, `CommitRequest`, `Assistant` and the assistant request structs), which breaks a crate that builds that struct with a struct literal. Each task updates every struct literal of the struct it changes in this repository, and its pull request carries a `BREAKING CHANGE:` footer. +Each task in this group adds a public field to an SDK struct (`CommitData`, `CommitRequest`, `Assistant` and the assistant request structs), which breaks a crate that builds that struct with a struct literal. Each task updates every struct literal of the struct it changes in this repository. Whether these pull requests and task 2.2's carry a `BREAKING CHANGE:` footer waits on [Open Question 9](design.md#9---should-the-sdk-field-additions-count-as-a-breaking-change). - [ ] 3.1 Add a commit read that sends `include=model` and returns `description`, `model_config` and `model_provider` to `sdk/src/prompts.rs`; verify with an httpmock test that asserts the query string and the parsed fields, and an integration test that reads a prompt with a model - [ ] 3.2 Add the commit's `id` to `CommitData` in `sdk/src/prompts.rs`, a call that lists a prompt's commits with their `id` and hash (`GET /api/v1/commits/{owner}/{repo}`), and calls that list, create and move commit tags, which name their commit by that `id` in `POST /api/v1/repos/{owner}/{repo}/tags` and `PATCH /api/v1/repos/{owner}/{repo}/tags/{tag_name}`; verify with httpmock tests that assert `commit_id` in each request body, and an integration test that pushes two commits, tags the first, finds the second commit's `id` through the commit list, moves the tag to it, and reads the tag back through the SDK From 85ed17dfed982f6bc95fc8caac83eed3d8b1b242 Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Tue, 13 Oct 2026 20:30:54 -0400 Subject: [PATCH 22/90] =?UTF-8?q?=F0=9F=93=9A=20docs(openspec):=20weigh=20?= =?UTF-8?q?non=5Fexhaustive=20and=20#815's=20choice=20in=20the=20SDK=20bre?= =?UTF-8?q?aking-change=20question?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit #807 Export, import, compare, and verify prompts and assistants across LangSmith environments Co-Authored-By: Claude Opus 5.5 --- openspec/changes/export-import-prompts-and-assistants/design.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/openspec/changes/export-import-prompts-and-assistants/design.md b/openspec/changes/export-import-prompts-and-assistants/design.md index 2f14ebb80..2a0a970a0 100644 --- a/openspec/changes/export-import-prompts-and-assistants/design.md +++ b/openspec/changes/export-import-prompts-and-assistants/design.md @@ -174,6 +174,6 @@ None. ### 9 - Should the SDK field additions count as a breaking change? > Context from the drafting agent, for question 9. -> Tasks 2.2 and 3.1 to 3.4 add public fields to `Deployment`, `CommitData`, `CommitRequest`, `Assistant` and the assistant request structs. A crate that builds one of those structs with a struct literal stops compiling, so by `docs/dev/git-scm-conventions.md` each of those pull requests would carry a `BREAKING CHANGE:` footer and the release would become a major version. `sdk/Cargo.toml` does not set `publish = false`, but the CLI in this workspace is the only user of the SDK that langstar knows of, and each pull request updates the struct literals in this repository. Recommendation: treat the additions as `feat` changes with no `BREAKING CHANGE:` footer, and name the new fields in the release notes. +> Tasks 2.2 and 3.1 to 3.4 add public fields to `Deployment`, `CommitData`, `CommitRequest`, `Assistant` and the assistant request structs. A crate that builds one of those structs with a struct literal stops compiling, so by `docs/dev/git-scm-conventions.md` each of those pull requests would carry a `BREAKING CHANGE:` footer and the release would become a major version. `sdk/Cargo.toml` does not set `publish = false`, but the CLI in this workspace is the only user of the SDK that langstar knows of, and each pull request updates the struct literals in this repository. A third option is to mark these structs `#[non_exhaustive]`: that is one breaking release now, and later field additions break nothing, but crates outside the SDK can then no longer build the structs with literals at all. [#815](https://github.com/codekiln/langstar/pull/815), the pull request for [#765](https://github.com/codekiln/langstar/issues/765), marks its SDK removals and return-type changes with `BREAKING CHANGE:` and leaves that choice to codekiln too, so one answer can set the policy for both. A removal or a changed return type breaks every caller; an added field breaks only a struct literal. Recommendation: treat the additions as `feat` changes with no `BREAKING CHANGE:` footer, and name the new fields in the release notes. From d42098b58e5fb0d2dc27c844b788509483d89af9 Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Tue, 13 Oct 2026 20:37:31 -0400 Subject: [PATCH 23/90] =?UTF-8?q?=F0=9F=93=9A=20docs(openspec):=20list=20C?= =?UTF-8?q?ommitManifestResponse=20among=20the=20changed=20SDK=20structs?= =?UTF-8?q?=20and=20give=20each=20rewritten=20sentence=20an=20actor?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds CommitManifestResponse, which PromptClient::get_commit returns, to the structs that question 9 and tasks.md group 3 name, and has task 3.1 extend it. Names the #815 link by its title, drops "a third option", links question 9 by its text, and limits prompt compare to the file's repo and commit objects. #807 Export, import, compare, and verify prompts and assistants across LangSmith environments Co-Authored-By: Claude Opus 5.5 --- .../export-import-prompts-and-assistants/design.md | 12 ++++++------ .../specs/prompt/spec.md | 2 +- .../export-import-prompts-and-assistants/tasks.md | 4 ++-- 3 files changed, 9 insertions(+), 9 deletions(-) diff --git a/openspec/changes/export-import-prompts-and-assistants/design.md b/openspec/changes/export-import-prompts-and-assistants/design.md index 2a0a970a0..75887ebfb 100644 --- a/openspec/changes/export-import-prompts-and-assistants/design.md +++ b/openspec/changes/export-import-prompts-and-assistants/design.md @@ -28,7 +28,7 @@ Langstar reads one API key and one workspace at a time, from flags, environment Export runs with the source environment's credentials and writes a file. Import runs later with the destination's credentials and reads it. Each run uses langstar's existing settings, so a user moving a prompt to another organization sets a different `LANGSMITH_API_KEY` for the import. Langstar needs no second set of credentials and no profiles. The file sits in Git between the two runs, so a reviewer sees the prompt or assistant change in a pull request. -The LangSmith Go CLI moves a prompt through a file the same way, as `references/langsmith-cli.md` shows. Langstar's export asks for the model, and the Go CLI's pull does not. Langstar's import reads the file export writes, while the Go CLI's push sends whatever JSON it reads as the manifest, so its own pull output reaches LangSmith wrapped in an envelope. Langstar's import also requires `--workspace-id`, so the user names the destination on the command line and a leftover environment variable never chooses it. +The LangSmith Go CLI moves a prompt through a file the same way, as `references/langsmith-cli.md` shows. Langstar's export asks for the model, and the Go CLI's pull does not. Langstar's import reads the file export writes, while the Go CLI's push sends whatever JSON it reads as the manifest, so its own pull output reaches LangSmith wrapped in an envelope. Langstar's import also requires `--workspace-id`, so langstar imports into the workspace the user typed on the command line, even when `LANGSMITH_WORKSPACE_ID` names another. ### The export file keeps the manifest LangSmith returned @@ -51,7 +51,7 @@ An assistant export has `kind: "assistant"`, a `source` with the deployment, ass The file holds no export time and langstar sorts its keys, so exporting an unchanged resource leaves Git with nothing to commit. -Import, compare and verify read a file only when its `format_version` is 1 and its `kind` matches the command: `prompt` for the prompt commands and `assistant` for the assistant commands. For any other file the command writes nothing, exits with status 1 and names the field and the value it found, so a file from a newer langstar, or an assistant file given to `prompt import`, stops the command before it changes anything. +Import, compare and verify read a file only when its `format_version` is 1 and its `kind` matches the command: `prompt` for the prompt commands and `assistant` for the assistant commands. For any other file the command writes nothing, exits with status 1 and names the field and the value it found. When a user runs `prompt import` on an assistant export, or on a file a newer langstar wrote, langstar stops with that error before it changes anything in the destination. ### Export asks LangSmith for the model @@ -91,7 +91,7 @@ An assistant refers to a prompt through whatever configuration key its graph rea ### compare, verify and import exit with 3 when they find a difference -Status 0 means the resource matches and every check passed. Status 3 means langstar found a difference or a failed check. Status 1 means any other error. Status 2 stays clap's: `Cli::parse()` in `run` in `cli/src/main.rs` exits with 2 when a user mistypes a flag or leaves out a required one, and this change leaves that alone, so a script never reads a typo as a difference. `prompt import` declares `--workspace-id` optional, as the other `langstar prompt` subcommands do, and checks for it itself, so a missing flag exits with status 1 and an error that names the flag. `diff` uses 1 for differences and 2 for errors; langstar keeps 1 for errors so that a script testing for failure sees the same status from every command. +Status 0 means the resource matches and every check passed. Status 3 means langstar found a difference or a failed check. Status 1 means any other error. Status 2 stays clap's: `Cli::parse()` in `run` in `cli/src/main.rs` exits with 2 when a user mistypes a flag or leaves out a required one, and this change leaves that alone, so a script never reads a typo as a difference. `prompt import` declares `--workspace-id` optional, as the other `langstar prompt` subcommands do, and checks for the flag itself. When a user leaves the flag out, langstar exits with status 1 and an error that names `--workspace-id`, where clap would have exited with 2. `diff` uses 1 for differences and 2 for errors; langstar keeps 1 for errors so that a script testing for failure sees the same status from every command. ### A Python script checks that an imported structured prompt still returns structured data @@ -106,7 +106,7 @@ A mise task runs a short Python script that pulls the imported prompt with `incl - [LangSmith may store a pushed model differently from the source] → Import reads the prompt back and reports any difference, and in [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8) a developer checks the round trip against the live API before anyone builds the commands. - [Pushing a manifest identical to the latest commit may return 409] → Import compares first and pushes nothing when the prompt matches. - [langstar finds a deployment's Agent Server URL only in `source_config.custom_url` (`resolve_deployment_url` in `cli/src/deployment_utils.rs`), and a deployment built from GitHub has none, so every `langstar assistant` command, the new ones included, fails on such a deployment] → In [export-import-prompts-and-assistants Task 2.2 Return the URL of a deployment built from GitHub](tasks.md#L13), a developer adds the control plane's top-level `url` to the SDK's `Deployment`, which drops it today, and makes `resolve_deployment_url` use it, before anyone builds the assistant export, compare, verify and import commands. -- [Tasks 2.2 and 3.1 to 3.4 add public fields to SDK structs, so a crate that builds one of those structs with a struct literal stops compiling] → Each of those pull requests updates every struct literal of the changed struct in this repository. [Open Question 9](#9---should-the-sdk-field-additions-count-as-a-breaking-change) asks whether they also carry a `BREAKING CHANGE:` footer, which `docs/dev/git-scm-conventions.md` says triggers a major version bump. +- [Tasks 2.2 and 3.1 to 3.4 add public fields to SDK structs, so a crate that builds one of those structs with a struct literal stops compiling] → Each of those pull requests updates every struct literal of the changed struct in this repository. [Should the SDK field additions count as a breaking change?](#9---should-the-sdk-field-additions-count-as-a-breaking-change) asks codekiln whether they also carry a `BREAKING CHANGE:` footer, which `docs/dev/git-scm-conventions.md` says makes the release a major version. - [A tag in the destination may already point at another commit] → `--update` moves the tag, and `--dry-run` lists the move. ## Resolved Questions @@ -153,7 +153,7 @@ None. ### 6 - Should prompt import refuse to run without --workspace-id even when LANGSMITH_WORKSPACE_ID is set? > Context from the drafting agent, for question 6. -> [#807 Export, import, compare, and verify prompts and assistants across LangSmith environments](https://github.com/codekiln/langstar/issues/807) asks for an explicit destination workspace. Requiring the flag means a leftover environment variable can never send a prompt to the wrong workspace, at the cost of one more flag in scripts. Assistant import already requires `--deployment`. Recommendation: require the flag. +> [#807 Export, import, compare, and verify prompts and assistants across LangSmith environments](https://github.com/codekiln/langstar/issues/807) asks for an explicit destination workspace. Requiring the flag means langstar imports into the workspace the user typed on the command line, even when `LANGSMITH_WORKSPACE_ID` names another, at the cost of one more flag in scripts. Assistant import already requires `--deployment`. Recommendation: require the flag. @@ -174,6 +174,6 @@ None. ### 9 - Should the SDK field additions count as a breaking change? > Context from the drafting agent, for question 9. -> Tasks 2.2 and 3.1 to 3.4 add public fields to `Deployment`, `CommitData`, `CommitRequest`, `Assistant` and the assistant request structs. A crate that builds one of those structs with a struct literal stops compiling, so by `docs/dev/git-scm-conventions.md` each of those pull requests would carry a `BREAKING CHANGE:` footer and the release would become a major version. `sdk/Cargo.toml` does not set `publish = false`, but the CLI in this workspace is the only user of the SDK that langstar knows of, and each pull request updates the struct literals in this repository. A third option is to mark these structs `#[non_exhaustive]`: that is one breaking release now, and later field additions break nothing, but crates outside the SDK can then no longer build the structs with literals at all. [#815](https://github.com/codekiln/langstar/pull/815), the pull request for [#765](https://github.com/codekiln/langstar/issues/765), marks its SDK removals and return-type changes with `BREAKING CHANGE:` and leaves that choice to codekiln too, so one answer can set the policy for both. A removal or a changed return type breaks every caller; an added field breaks only a struct literal. Recommendation: treat the additions as `feat` changes with no `BREAKING CHANGE:` footer, and name the new fields in the release notes. +> Tasks 2.2 and 3.1 to 3.4 add public fields to `Deployment`, `CommitManifestResponse` (which `PromptClient::get_commit` returns), `CommitData`, `CommitRequest`, `Assistant` and the assistant request structs. A crate that builds one of those structs with a struct literal stops compiling, so by `docs/dev/git-scm-conventions.md` each of those pull requests would carry a `BREAKING CHANGE:` footer and the release would become a major version. `sdk/Cargo.toml` does not set `publish = false`, but the CLI in this workspace is the only user of the SDK that langstar knows of, and each pull request updates the struct literals in this repository. Marking the structs `#[non_exhaustive]` would break outside crates once, in this release, and let later fields land without breaking anything, but crates outside the SDK could then no longer build the structs with literals at all. [🚨 BREAKING CHANGE(sdk): move annotation-queue run calls to the platform items API · Pull Request #815](https://github.com/codekiln/langstar/pull/815), the pull request for [#765](https://github.com/codekiln/langstar/issues/765), already marks its SDK changes as breaking and leaves that choice to codekiln too, so one answer to this question can set the policy for both pull requests. A removal or a changed return type breaks every caller; an added field breaks only a struct literal. Recommendation: treat the additions as `feat` changes with no `BREAKING CHANGE:` footer, and name the new fields in the release notes. diff --git a/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md index bbcaf3c23..8560650c3 100644 --- a/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md +++ b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md @@ -172,7 +172,7 @@ After writing, `langstar prompt import` SHALL record the commit hash in the dest ### Requirement: Compare a prompt with its export file -`langstar prompt compare ` SHALL read the prompt with the file's handle from the workspace, with the model included, and its repo details, and list each difference by its JSON path with the file's value and the workspace's value. `--commit ` SHALL choose the commit; without it langstar SHALL read the latest commit. It SHALL compare only the fields the file holds, leaving out server-set IDs and timestamps, and every key in the manifest, its `id` arrays included. +`langstar prompt compare ` SHALL read the prompt with the file's handle from the workspace, with its model and repo details, and list each difference by its JSON path with both values. `--commit ` SHALL choose the commit, and without it langstar SHALL read the latest. It SHALL compare every field in the file's `repo` and `commit`, including every manifest key and its `id` arrays. Those hold no IDs or timestamps LangSmith sets, so compare never reports them. #### Scenario: Prompt matches diff --git a/openspec/changes/export-import-prompts-and-assistants/tasks.md b/openspec/changes/export-import-prompts-and-assistants/tasks.md index 31dde9124..e2abc0837 100644 --- a/openspec/changes/export-import-prompts-and-assistants/tasks.md +++ b/openspec/changes/export-import-prompts-and-assistants/tasks.md @@ -15,9 +15,9 @@ Each numbered group below is one pull request, merged in the order listed. Task ## 3. SDK reads and writes the fields export and import need -Each task in this group adds a public field to an SDK struct (`CommitData`, `CommitRequest`, `Assistant` and the assistant request structs), which breaks a crate that builds that struct with a struct literal. Each task updates every struct literal of the struct it changes in this repository. Whether these pull requests and task 2.2's carry a `BREAKING CHANGE:` footer waits on [Open Question 9](design.md#9---should-the-sdk-field-additions-count-as-a-breaking-change). +Each task in this group adds a public field to an SDK struct (`CommitManifestResponse`, which `PromptClient::get_commit` returns, `CommitData`, `CommitRequest`, `Assistant` and the assistant request structs), which breaks a crate that builds that struct with a struct literal. Each task updates every struct literal of the struct it changes in this repository. Whether these pull requests and task 2.2's carry a `BREAKING CHANGE:` footer waits on [Should the SDK field additions count as a breaking change?](design.md#9---should-the-sdk-field-additions-count-as-a-breaking-change). -- [ ] 3.1 Add a commit read that sends `include=model` and returns `description`, `model_config` and `model_provider` to `sdk/src/prompts.rs`; verify with an httpmock test that asserts the query string and the parsed fields, and an integration test that reads a prompt with a model +- [ ] 3.1 Add `description`, `model_config` and `model_provider` to `CommitManifestResponse`, and a commit read that sends `include=model` and returns that struct, to `sdk/src/prompts.rs`; verify with an httpmock test that asserts the query string and the parsed fields, and an integration test that reads a prompt with a model - [ ] 3.2 Add the commit's `id` to `CommitData` in `sdk/src/prompts.rs`, a call that lists a prompt's commits with their `id` and hash (`GET /api/v1/commits/{owner}/{repo}`), and calls that list, create and move commit tags, which name their commit by that `id` in `POST /api/v1/repos/{owner}/{repo}/tags` and `PATCH /api/v1/repos/{owner}/{repo}/tags/{tag_name}`; verify with httpmock tests that assert `commit_id` in each request body, and an integration test that pushes two commits, tags the first, finds the second commit's `id` through the commit list, moves the tag to it, and reads the tag back through the SDK - [ ] 3.3 Add `context` and `version` to `Assistant`, `context` and `graph_id` to `UpdateAssistantRequest`, `context` to `CreateAssistantRequest`, and a `graph_id` filter to `AssistantSearchRequest`, in `sdk/src/assistants.rs`; verify with httpmock tests of the request bodies and an integration test that creates an assistant with context, reads it back with its version, finds it by graph ID, and updates its graph ID and reads the new graph ID back - [ ] 3.4 Add `description` to `CommitRequest`, and a repo update that sends `PATCH /api/v1/repos/{owner}/{repo}` with `description`, `readme`, `tags` and `is_public`, to `sdk/src/prompts.rs`; verify with httpmock tests of the request bodies and an integration test that pushes a commit with a description, updates the repo's description, readme, tags and visibility, and reads each back From 701fea51a507832b52ffe77b40b4931d96f52986 Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Tue, 13 Oct 2026 20:40:40 -0400 Subject: [PATCH 24/90] =?UTF-8?q?=F0=9F=93=9A=20docs(openspec):=20have=20p?= =?UTF-8?q?rompt=20import=20and=20verify=20check=20requires=20against=20th?= =?UTF-8?q?e=20manifest?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Someone can edit an export file's manifest in Git and leave requires.secrets as it was, so import and verify work out both requires lists again from the manifest and stop with status 1 when they differ. Adds a scenario and a verify test for it, and names CommitManifestResponse in the proposal's impact list. #807 Export, import, compare, and verify prompts and assistants across LangSmith environments Co-Authored-By: Claude Opus 5.5 --- .../export-import-prompts-and-assistants/design.md | 2 +- .../export-import-prompts-and-assistants/proposal.md | 2 +- .../specs/prompt/spec.md | 8 +++++++- .../changes/export-import-prompts-and-assistants/tasks.md | 2 +- 4 files changed, 10 insertions(+), 4 deletions(-) diff --git a/openspec/changes/export-import-prompts-and-assistants/design.md b/openspec/changes/export-import-prompts-and-assistants/design.md index 75887ebfb..dfbcb9ebe 100644 --- a/openspec/changes/export-import-prompts-and-assistants/design.md +++ b/openspec/changes/export-import-prompts-and-assistants/design.md @@ -75,7 +75,7 @@ Compare reads the destination into the export file's `repo`, `commit` and `assis ### Verify checks what the copy needs to run -Prompt verify checks that the workspace holds every secret the prompt's model reads. Export lists those names in `requires.secrets` by walking the manifest for LangChain secret references, `{"lc": 1, "type": "secret", "id": ["OPENAI_API_KEY"]}`, the shape the Python SDK's `_manifest_has_secrets` matches. Verify looks each name up among the workspace's secrets (`GET /api/v1/workspaces/current/secrets`, which `langstar secrets list` already calls) and prints the name of each missing secret and no secret value. +Prompt verify checks that the workspace holds every secret the prompt's model reads. Export lists those names in `requires.secrets` by walking the manifest for LangChain secret references, `{"lc": 1, "type": "secret", "id": ["OPENAI_API_KEY"]}`, the shape the Python SDK's `_manifest_has_secrets` matches. Verify looks each name up among the workspace's secrets (`GET /api/v1/workspaces/current/secrets`, which `langstar secrets list` already calls) and prints the name of each missing secret and no secret value. Someone can edit the manifest in Git and forget `requires`, so import and verify walk the manifest again and stop with status 1, naming the field, when either list differs from the file's. Prompt verify also notes when the workspace has no saved model configuration for the prompt's model, and leaves the exit status alone, because the prompt runs with the model its manifest carries. Export takes the model's class name, such as `ChatOpenAI`, from the last element of the chat model constructor's `id` in the manifest, such as `["langchain", "chat_models", "openai", "ChatOpenAI"]`, and stores it in `requires.model_provider`. Verify looks for a saved model configuration (`GET /api/v1/playground-settings`, which `langstar model-config list` calls) whose settings `id` ends in that class name. LangSmith also returns a `model_provider` field beside the manifest, which the file keeps as `commit.model_provider`; no SDK or spec shows what it holds, so in [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8) a developer records the value the live API returns. [#807 Export, import, compare, and verify prompts and assistants across LangSmith environments](https://github.com/codekiln/langstar/issues/807) asks for the note, and [Should prompt verify report a missing saved model configuration?](#7---should-prompt-verify-report-a-missing-saved-model-configuration) asks whether to keep it. diff --git a/openspec/changes/export-import-prompts-and-assistants/proposal.md b/openspec/changes/export-import-prompts-and-assistants/proposal.md index 510157af8..f679f5199 100644 --- a/openspec/changes/export-import-prompts-and-assistants/proposal.md +++ b/openspec/changes/export-import-prompts-and-assistants/proposal.md @@ -30,7 +30,7 @@ A prompt belongs to a workspace and an assistant belongs to a deployment, but bo ## Impact -- SDK: `sdk/src/prompts.rs` gains a commit read that asks for the model, a commit description on commit create, the new commit's `id`, a repo update, a commit list that returns each commit's `id`, and calls that list, create and move commit tags. `sdk/src/assistants.rs` gains the `context` and `version` fields, `graph_id` on update, and a search by graph ID. `sdk/src/deployments.rs` gains the deployment's `url`. Each new public field breaks a crate outside this repository that builds that SDK struct with a struct literal; design.md asks codekiln whether that makes the release a major version. +- SDK: `sdk/src/prompts.rs` gains a commit read that asks for the model and returns it on `CommitManifestResponse`, a commit description on commit create, the new commit's `id`, a repo update, a commit list that returns each commit's `id`, and calls that list, create and move commit tags. `sdk/src/assistants.rs` gains the `context` and `version` fields, `graph_id` on update, and a search by graph ID. `sdk/src/deployments.rs` gains the deployment's `url`. Each new public field breaks a crate outside this repository that builds that SDK struct with a struct literal; design.md asks codekiln whether that makes the release a major version. - CLI: new subcommands in `cli/src/commands/prompt.rs` and `cli/src/commands/assistant.rs`, and a shared module for the export file, the destination record, comparison and the exit status. `resolve_deployment_url` in `cli/src/deployment_utils.rs` also returns the Agent Server URL of a deployment built from GitHub, which every `langstar assistant` command needs. - Tests need a second workspace, a second deployment and one model provider key for the structured output check. codekiln creates the workspace and the key, and langstar's tests read them through `fnox.toml` and the CI workflow; the `TestDeploymentConfig` fixture creates the deployment on the first run and the tests reuse it after that. - Docs: a new usage page on moving prompts and assistants between environments. diff --git a/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md index 8560650c3..e8edce179 100644 --- a/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md +++ b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md @@ -45,7 +45,7 @@ Exporting the same commit twice SHALL produce identical files. The file SHALL ho ### Requirement: List the credentials a prompt needs without their values -The export file SHALL list, under `requires.secrets`, the name of every secret the manifest refers to, and under `requires.model_provider`, the class name at the end of the chat model constructor's `id` in the manifest, such as `ChatOpenAI`. It SHALL hold no secret values. +The export file SHALL list, under `requires.secrets`, the name of every secret the manifest refers to, and under `requires.model_provider`, the class name at the end of the chat model constructor's `id` in the manifest, such as `ChatOpenAI`. It SHALL hold no secret values. Import and verify SHALL work out both lists again from the file's manifest, and when either differs from the file's `requires`, write nothing, name the field and exit with status 1. #### Scenario: Model reads an API key secret @@ -53,6 +53,12 @@ The export file SHALL list, under `requires.secrets`, the name of every secret t - **THEN** `requires.secrets` holds `OPENAI_API_KEY` - **AND** no value of that secret appears in the file +#### Scenario: Manifest edited without its requires list + +- **WHEN** someone edits the file's manifest so its model refers to the secret `ANTHROPIC_API_KEY`, and leaves `requires.secrets` holding only `OPENAI_API_KEY` +- **AND** a user runs `langstar prompt import my-prompt.json --workspace-id ` +- **THEN** langstar creates no prompt, names `requires.secrets` and exits with status 1 + ### Requirement: Import names its destination workspace `langstar prompt import ` SHALL require `--workspace-id`. It SHALL refuse to run when the flag is missing, even when `LANGSMITH_WORKSPACE_ID` is set, and exit with status 1 and an error naming the flag. diff --git a/openspec/changes/export-import-prompts-and-assistants/tasks.md b/openspec/changes/export-import-prompts-and-assistants/tasks.md index e2abc0837..93f281b4f 100644 --- a/openspec/changes/export-import-prompts-and-assistants/tasks.md +++ b/openspec/changes/export-import-prompts-and-assistants/tasks.md @@ -37,7 +37,7 @@ Each task in this group adds a public field to an SDK struct (`CommitManifestRes ## 6. prompt compare and verify - [ ] 6.1 Add `langstar prompt compare`; verify with integration tests that export a prompt, change its model temperature through the SDK, and assert the reported path, both values and exit status 3, and that a changed commit description and a changed repo readme are each reported the same way, and CLI tests that `langstar -f json prompt compare` prints one JSON object with `status`, `differences` and `checks` to stdout and its progress to stderr, and that `prompt compare` with a mistyped flag exits with status 2, and that `prompt compare` of a file holding `"format_version": 2` exits with status 1 and names `format_version` and the value 2 -- [ ] 6.2 Add `langstar prompt verify` with the secret check and the saved model configuration note, matching `requires.model_provider` against the class name at the end of each configuration's settings `id`; verify with integration tests against a workspace that lacks a listed secret and one that has it, and a test that a missing saved model configuration is reported and leaves the exit status at 0 +- [ ] 6.2 Add `langstar prompt verify` with the secret check and the saved model configuration note, matching `requires.model_provider` against the class name at the end of each configuration's settings `id`; verify with integration tests against a workspace that lacks a listed secret and one that has it, a test that a missing saved model configuration is reported and leaves the exit status at 0, and a test that a file whose manifest refers to a secret its `requires.secrets` leaves out exits with status 1 and names `requires.secrets` - [ ] 6.3 Document compare and verify; verify that each documented command runs as written ## 7. prompt import From 4ea630c166377a1afdaef2981e43b7e4ae7aa5b7 Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Tue, 13 Oct 2026 20:54:00 -0400 Subject: [PATCH 25/90] =?UTF-8?q?=F0=9F=93=9A=20docs(openspec):=20merge=20?= =?UTF-8?q?the=20destination=20graph=20config=20to=20main=20before=20its?= =?UTF-8?q?=20fixture=20and=20test=20import's=20requires=20check?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Task 2.3 now adds the destination test graph's langgraph.json in a pull request of its own, because the control plane builds test deployments from main; the fixture moves to 2.4 with a name prefix of its own. Task 7.1 tests import against an edited manifest. Question 9's context leads with its argument, the SDK risk names its structs, and the compare paragraph says what compare lists. #807 Export, import, compare, and verify prompts and assistants across LangSmith environments Co-Authored-By: Claude Opus 5.5 --- .../export-import-prompts-and-assistants/design.md | 8 ++++---- .../export-import-prompts-and-assistants/proposal.md | 2 +- .../export-import-prompts-and-assistants/tasks.md | 9 +++++---- 3 files changed, 10 insertions(+), 9 deletions(-) diff --git a/openspec/changes/export-import-prompts-and-assistants/design.md b/openspec/changes/export-import-prompts-and-assistants/design.md index dfbcb9ebe..a65fe8d54 100644 --- a/openspec/changes/export-import-prompts-and-assistants/design.md +++ b/openspec/changes/export-import-prompts-and-assistants/design.md @@ -71,7 +71,7 @@ Without `--update`, import lists the differences and exits with status 3. `--dry ### Compare reports differences by JSON path -Compare reads the destination into the export file's `repo`, `commit` and `assistant` objects, which hold none of the fields the server sets (IDs, version numbers, timestamps, owner and like counts), and walks both JSON documents. Compare drops no key by name inside those objects, so the `id` arrays inside a manifest, which name the prompt and model classes such as `ChatOpenAI`, count as differences. The source environment's workspace, deployment and IDs sit in `source`, so they never show up as differences against a copy. Each difference names its JSON Pointer path and both values. For a prompt, the comparison covers the commit description, the commit tags and the commit each one points at, and the repo's description, readme, tags and visibility as well as the manifest. Import runs the same comparison after it writes, so a field LangSmith or the Agent Server drops shows up as a difference. +Compare reads the prompt or assistant from the destination, puts it in the shape of the export file's `repo`, `commit` and `assistant` objects, and lists every value that differs from the file. Those objects hold no field the server sets, such as IDs, version numbers, timestamps, owner and like counts, so compare checks every key in them, including the `id` arrays inside a manifest that name the prompt and model classes, such as `ChatOpenAI`. The source environment's workspace, deployment and IDs sit in `source`, so they never show up as differences against a copy. Each difference names its JSON Pointer path and both values. For a prompt, the comparison covers the commit description, the commit tags and the commit each one points at, and the repo's description, readme, tags and visibility as well as the manifest. Import runs the same comparison after it writes, so a field LangSmith or the Agent Server drops shows up as a difference. ### Verify checks what the copy needs to run @@ -106,7 +106,7 @@ A mise task runs a short Python script that pulls the imported prompt with `incl - [LangSmith may store a pushed model differently from the source] → Import reads the prompt back and reports any difference, and in [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8) a developer checks the round trip against the live API before anyone builds the commands. - [Pushing a manifest identical to the latest commit may return 409] → Import compares first and pushes nothing when the prompt matches. - [langstar finds a deployment's Agent Server URL only in `source_config.custom_url` (`resolve_deployment_url` in `cli/src/deployment_utils.rs`), and a deployment built from GitHub has none, so every `langstar assistant` command, the new ones included, fails on such a deployment] → In [export-import-prompts-and-assistants Task 2.2 Return the URL of a deployment built from GitHub](tasks.md#L13), a developer adds the control plane's top-level `url` to the SDK's `Deployment`, which drops it today, and makes `resolve_deployment_url` use it, before anyone builds the assistant export, compare, verify and import commands. -- [Tasks 2.2 and 3.1 to 3.4 add public fields to SDK structs, so a crate that builds one of those structs with a struct literal stops compiling] → Each of those pull requests updates every struct literal of the changed struct in this repository. [Should the SDK field additions count as a breaking change?](#9---should-the-sdk-field-additions-count-as-a-breaking-change) asks codekiln whether they also carry a `BREAKING CHANGE:` footer, which `docs/dev/git-scm-conventions.md` says makes the release a major version. +- [The SDK changes for export and import add public fields to `Deployment`, `CommitManifestResponse`, `CommitData`, `CommitRequest`, `Assistant` and the assistant request structs, so a crate that builds one of those structs with a struct literal stops compiling] → Each pull request that adds a field updates every struct literal of that struct in this repository. [Should the SDK field additions count as a breaking change?](#9---should-the-sdk-field-additions-count-as-a-breaking-change) asks codekiln whether those pull requests also carry a `BREAKING CHANGE:` footer, which `docs/dev/git-scm-conventions.md` says makes the release a major version. - [A tag in the destination may already point at another commit] → `--update` moves the tag, and `--dry-run` lists the move. ## Resolved Questions @@ -146,7 +146,7 @@ None. ### 5 - Which second workspace and second deployment should the tests use? > Context from the drafting agent, for question 5. -> The tests need a destination workspace and a destination deployment. The draft adds `LANGSTAR_TEST_DEST_WORKSPACE_ID`, a second workspace in the test organization so the existing API key works, and a second test deployment that the existing `TestDeploymentConfig` fixture builds from its own `langgraph.json`, because the control plane allows one deployment per repository and graph config file. A second deployment costs build time and hosting in every CI run that creates it. Recommendation: a second workspace in the same organization, which codekiln creates, and one destination deployment that the `TestDeploymentConfig` fixture creates on the first run and the tests reuse after that. +> The tests need a destination workspace and a destination deployment. The draft adds `LANGSTAR_TEST_DEST_WORKSPACE_ID`, a second workspace in the test organization so the existing API key works, and a second test deployment that the destination deployment fixture from [export-import-prompts-and-assistants Task 2.4 Add a destination deployment fixture](tasks.md#L15) creates from its own `langgraph.json` and its own name prefix, because the control plane allows one deployment per repository and graph config file and `find_reusable_deployment` matches by name prefix first. The control plane builds test deployments from `main`, so that `langgraph.json` merges to `main` in a pull request of its own, [export-import-prompts-and-assistants Task 2.3 Add the destination test graph's `langgraph.json`](tasks.md#L14), first. A second deployment costs build time and hosting in every CI run that creates it. Recommendation: a second workspace in the same organization, which codekiln creates, and one destination deployment that the destination deployment fixture from task 2.4 creates on the first run and the tests reuse after that. @@ -174,6 +174,6 @@ None. ### 9 - Should the SDK field additions count as a breaking change? > Context from the drafting agent, for question 9. -> Tasks 2.2 and 3.1 to 3.4 add public fields to `Deployment`, `CommitManifestResponse` (which `PromptClient::get_commit` returns), `CommitData`, `CommitRequest`, `Assistant` and the assistant request structs. A crate that builds one of those structs with a struct literal stops compiling, so by `docs/dev/git-scm-conventions.md` each of those pull requests would carry a `BREAKING CHANGE:` footer and the release would become a major version. `sdk/Cargo.toml` does not set `publish = false`, but the CLI in this workspace is the only user of the SDK that langstar knows of, and each pull request updates the struct literals in this repository. Marking the structs `#[non_exhaustive]` would break outside crates once, in this release, and let later fields land without breaking anything, but crates outside the SDK could then no longer build the structs with literals at all. [🚨 BREAKING CHANGE(sdk): move annotation-queue run calls to the platform items API · Pull Request #815](https://github.com/codekiln/langstar/pull/815), the pull request for [#765](https://github.com/codekiln/langstar/issues/765), already marks its SDK changes as breaking and leaves that choice to codekiln too, so one answer to this question can set the policy for both pull requests. A removal or a changed return type breaks every caller; an added field breaks only a struct literal. Recommendation: treat the additions as `feat` changes with no `BREAKING CHANGE:` footer, and name the new fields in the release notes. +> A new public field breaks only a crate that builds the struct with a struct literal. The only crate known to use the SDK is langstar's own CLI, and each pull request that adds a field updates the CLI's struct literals. `sdk/Cargo.toml` leaves the crate publishable, though, so an outside crate could break, and `docs/dev/git-scm-conventions.md` gives every API break a `BREAKING CHANGE:` footer, which makes the release a major version. The structs that gain fields are `Deployment`, `CommitManifestResponse` (which `PromptClient::get_commit` returns), `CommitData`, `CommitRequest`, `Assistant` and the assistant request structs. Marking them `#[non_exhaustive]` would stop outside crates from building them with struct literals at all, which breaks those crates once, in this release, and after that a new field would break nothing. [🚨 BREAKING CHANGE(sdk): move annotation-queue run calls to the platform items API · Pull Request #815](https://github.com/codekiln/langstar/pull/815) removes SDK functions and changes return types, marks itself breaking, and leaves that choice to codekiln too, so one answer to this question can set the policy for both pull requests. Recommendation: treat the additions as `feat` changes with no `BREAKING CHANGE:` footer, and name the new fields in the release notes. diff --git a/openspec/changes/export-import-prompts-and-assistants/proposal.md b/openspec/changes/export-import-prompts-and-assistants/proposal.md index f679f5199..f42b698af 100644 --- a/openspec/changes/export-import-prompts-and-assistants/proposal.md +++ b/openspec/changes/export-import-prompts-and-assistants/proposal.md @@ -32,7 +32,7 @@ A prompt belongs to a workspace and an assistant belongs to a deployment, but bo - SDK: `sdk/src/prompts.rs` gains a commit read that asks for the model and returns it on `CommitManifestResponse`, a commit description on commit create, the new commit's `id`, a repo update, a commit list that returns each commit's `id`, and calls that list, create and move commit tags. `sdk/src/assistants.rs` gains the `context` and `version` fields, `graph_id` on update, and a search by graph ID. `sdk/src/deployments.rs` gains the deployment's `url`. Each new public field breaks a crate outside this repository that builds that SDK struct with a struct literal; design.md asks codekiln whether that makes the release a major version. - CLI: new subcommands in `cli/src/commands/prompt.rs` and `cli/src/commands/assistant.rs`, and a shared module for the export file, the destination record, comparison and the exit status. `resolve_deployment_url` in `cli/src/deployment_utils.rs` also returns the Agent Server URL of a deployment built from GitHub, which every `langstar assistant` command needs. -- Tests need a second workspace, a second deployment and one model provider key for the structured output check. codekiln creates the workspace and the key, and langstar's tests read them through `fnox.toml` and the CI workflow; the `TestDeploymentConfig` fixture creates the deployment on the first run and the tests reuse it after that. +- Tests need a second workspace, a second deployment and one model provider key for the structured output check. codekiln creates the workspace and the key, and langstar's tests read them through `fnox.toml` and the CI workflow; the destination deployment fixture from [export-import-prompts-and-assistants Task 2.4 Add a destination deployment fixture](tasks.md#L15) creates the deployment on the first run and the tests reuse it after that. - Docs: a new usage page on moving prompts and assistants between environments. ## Citations diff --git a/openspec/changes/export-import-prompts-and-assistants/tasks.md b/openspec/changes/export-import-prompts-and-assistants/tasks.md index 93f281b4f..589f3f488 100644 --- a/openspec/changes/export-import-prompts-and-assistants/tasks.md +++ b/openspec/changes/export-import-prompts-and-assistants/tasks.md @@ -1,6 +1,6 @@ # Tasks -Each numbered group below is one pull request, merged in the order listed. Task 2.1, which adds the destination workspace and the model provider key, waits on codekiln to create that workspace and an OpenAI key, so it gets a pull request of its own and the rest of group 2 goes ahead without it. That pull request merges before group 6, prompt compare and verify, whose tests are the first to use the destination workspace. +Each numbered group below is one pull request, merged in the order listed. Task 2.1, which adds the destination workspace and the model provider key, waits on codekiln to create that workspace and an OpenAI key, so it gets a pull request of its own and the rest of group 2 goes ahead without it. Task 2.3, the destination test graph's `langgraph.json`, also gets a pull request of its own, because the control plane builds test deployments from `main` and task 2.4's fixture needs that file there. That pull request merges before group 6, prompt compare and verify, whose tests are the first to use the destination workspace. ## 1. Check the API round trip before building on it @@ -11,11 +11,12 @@ Each numbered group below is one pull request, merged in the order listed. Task - [ ] 2.1 Add the destination workspace and the model provider key to `fnox.toml`, the CI workflow and `docs/dev/environment-variables.md`, with placeholders only; verify with `mise run check` and a CI run that reads both. This task waits on codekiln to create a second workspace in the test organization and an OpenAI API key, and to store both in 1Password and the repository's CI secrets and the key as the destination workspace's `OPENAI_API_KEY` secret. The import tests need the workspace as a place to import into, and its secret lets prompt import's secret check pass for a prompt with an OpenAI model. The key lets the check in 7.3 run each imported structured prompt once against a small OpenAI model, which bills that account on every CI run; it is the one test that shows a copy still returns structured data, where every other test compares text. - [ ] 2.2 Add `url` to `Deployment` in `sdk/src/deployments.rs`, with a method that returns `url` and falls back to `source_config.custom_url`, and make `resolve_deployment_url` in `cli/src/deployment_utils.rs` call that method, so that a deployment built from GitHub resolves; end the four `Deployment` struct literals in `sdk/tests/integration_deployment_workflow.rs` and `cli/src/commands/deployment.rs` with `..Default::default()`, because a crate that builds a `Deployment` with a struct literal stops compiling when the field is added; add a unit test for each case in `sdk/src/deployments.rs`, and verify by removing `#[ignore]` from `test_assistant_create_basic` in `cli/tests/assistant_command_test.rs` and running it against the test deployment -- [ ] 2.3 Add a destination deployment fixture to `sdk/src/test_utils.rs` and `cli/tests/common/fixtures.rs` that builds the test graph from its own `langgraph.json` under `tests/fixtures/`, because the control plane allows one deployment per repository and graph config file and `find_reusable_deployment` would otherwise hand back the source deployment; verify with an integration test that asserts the destination and source deployment IDs differ and lists assistants on the destination deployment through the URL that 2.2 resolves +- [ ] 2.3 Add the destination test graph's own `langgraph.json` under `tests/fixtures/`, naming the same test graph as `tests/fixtures/test-graph-deployment/langgraph.json`, in a pull request of its own that merges to `main` before the pull request for 2.4, because the control plane builds every test deployment from `main` (`TestDeploymentConfig::default` sets `branch` to `main`); verify by running `langstar deployment create` with `--branch` set to this pull request's branch and `--config-path` set to the new file, listing the test graph's assistants on that deployment, and deleting it +- [ ] 2.4 Add a destination deployment fixture to `sdk/src/test_utils.rs` and `cli/tests/common/fixtures.rs` with its own name prefix, one that does not start with `pr-integration-test`, and the `langgraph.json` from 2.3. `find_reusable_deployment` returns any deployment whose name starts with the fixture's prefix before it checks the graph config, and the control plane allows one deployment per repository and graph config file, so a shared prefix or a shared `langgraph.json` would hand back the source deployment, and a prefix starting with `pr-integration-test` would hand the destination deployment to the source fixture; verify with an integration test that asserts the destination and source deployment IDs differ and lists assistants on the destination deployment through the URL that 2.2 resolves ## 3. SDK reads and writes the fields export and import need -Each task in this group adds a public field to an SDK struct (`CommitManifestResponse`, which `PromptClient::get_commit` returns, `CommitData`, `CommitRequest`, `Assistant` and the assistant request structs), which breaks a crate that builds that struct with a struct literal. Each task updates every struct literal of the struct it changes in this repository. Whether these pull requests and task 2.2's carry a `BREAKING CHANGE:` footer waits on [Should the SDK field additions count as a breaking change?](design.md#9---should-the-sdk-field-additions-count-as-a-breaking-change). +Each task in this group adds a public field to an SDK struct (`CommitManifestResponse`, which `PromptClient::get_commit` returns, `CommitData`, `CommitRequest`, `Assistant` and the assistant request structs), which breaks a crate that builds that struct with a struct literal. Each task updates every struct literal of the struct it changes in this repository. Whether these pull requests and the one for task 2.2, which adds `url` to `Deployment`, carry a `BREAKING CHANGE:` footer waits on [Should the SDK field additions count as a breaking change?](design.md#9---should-the-sdk-field-additions-count-as-a-breaking-change). - [ ] 3.1 Add `description`, `model_config` and `model_provider` to `CommitManifestResponse`, and a commit read that sends `include=model` and returns that struct, to `sdk/src/prompts.rs`; verify with an httpmock test that asserts the query string and the parsed fields, and an integration test that reads a prompt with a model - [ ] 3.2 Add the commit's `id` to `CommitData` in `sdk/src/prompts.rs`, a call that lists a prompt's commits with their `id` and hash (`GET /api/v1/commits/{owner}/{repo}`), and calls that list, create and move commit tags, which name their commit by that `id` in `POST /api/v1/repos/{owner}/{repo}/tags` and `PATCH /api/v1/repos/{owner}/{repo}/tags/{tag_name}`; verify with httpmock tests that assert `commit_id` in each request body, and an integration test that pushes two commits, tags the first, finds the second commit's `id` through the commit list, moves the tag to it, and reads the tag back through the SDK @@ -42,7 +43,7 @@ Each task in this group adds a public field to an SDK struct (`CommitManifestRes ## 7. prompt import -- [ ] 7.1 Add `langstar prompt import`, which exits with status 1 and names `--workspace-id` when the flag is missing, runs the secret and saved model configuration checks of `prompt verify` from 6.2 before it writes, creates the repo, the commit and its tags, then reads the commit back; verify with integration tests that export from the test workspace, import into the destination workspace, and compare the full source and destination commits through the SDK with the model included, the commit description among them, and the repo's description, readme, tags and visibility, for every prompt shape from 5.1, that a file whose `requires.secrets` names a secret the destination workspace lacks creates no prompt, names the secret and exits with status 3, and a CLI test that runs import without `--workspace-id` while `LANGSMITH_WORKSPACE_ID` is set and asserts that it exits with status 1, names the missing flag and creates no prompt, and a CLI test that `prompt import` of an assistant export exits with status 1, names `kind` and the value `assistant`, and creates no prompt +- [ ] 7.1 Add `langstar prompt import`, which exits with status 1 and names `--workspace-id` when the flag is missing, runs the secret and saved model configuration checks of `prompt verify` from 6.2 before it writes, creates the repo, the commit and its tags, then reads the commit back; verify with integration tests that export from the test workspace, import into the destination workspace, and compare the full source and destination commits through the SDK with the model included, the commit description among them, and the repo's description, readme, tags and visibility, for every prompt shape from 5.1, that a file whose `requires.secrets` names a secret the destination workspace lacks creates no prompt, names the secret and exits with status 3, and a CLI test that runs import without `--workspace-id` while `LANGSMITH_WORKSPACE_ID` is set and asserts that it exits with status 1, names the missing flag and creates no prompt, and a CLI test that `prompt import` of an assistant export exits with status 1, names `kind` and the value `assistant`, and creates no prompt, and a CLI test that imports a file whose manifest refers to `ANTHROPIC_API_KEY` while `requires.secrets` holds only `OPENAI_API_KEY`, and another whose `requires.model_provider` names a class other than the one at the end of the manifest's chat model `id`, asserting that each creates no prompt, names the field and exits with status 1 - [ ] 7.2 Add `--update`, `--dry-run`, `--record` and the destination record; verify with integration tests that a repeated import creates no commit, that a changed destination is refused without `--update`, that an import after someone pushed a commit to the destination since the recorded import says the destination changed since the last import and exits with status 3, that `--update` adds one commit with the right parent and the file's commit description, that `--update` against a destination whose repo description, readme, tags or visibility differ while the manifest matches updates the repo and adds no commit, that `--update` against a destination whose `prod` tag points at an earlier commit while the manifest matches moves the tag and adds no commit, that `--dry-run` lists the move, and that `--dry-run` writes nothing, and an httpmock test in which the commit read back lacks a field the file holds, asserting that import names that field's JSON Pointer path, exits with status 3, and leaves the new commit hash in the destination record - [ ] 7.3 Add the structured output check as a mise task that pulls the imported prompt with `include_model=True` and a `secrets` map holding the model provider key read from the environment, invokes it and validates the result against the schema; verify by running it from the import integration test for both methods - [ ] 7.4 Document prompt import, the workspace scope of prompts and setting up destination credentials; verify that each documented command runs as written From 41a7787602f96493592a051d7670794eb4f25fae Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Tue, 13 Oct 2026 21:08:20 -0400 Subject: [PATCH 26/90] =?UTF-8?q?=F0=9F=93=9A=20docs(openspec):=20say=20th?= =?UTF-8?q?at=20non=5Fexhaustive=20breaks=20langstar's=20own=20CLI,=20orde?= =?UTF-8?q?r=20task=202.3's=20pull=20request=20before=20group=202,=20and?= =?UTF-8?q?=20shorten=20question=205's=20context?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit #807 Export, import, compare, and verify prompts and assistants across LangSmith environments Co-Authored-By: Claude Opus 5.5 --- .../changes/export-import-prompts-and-assistants/design.md | 4 ++-- .../changes/export-import-prompts-and-assistants/tasks.md | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/openspec/changes/export-import-prompts-and-assistants/design.md b/openspec/changes/export-import-prompts-and-assistants/design.md index a65fe8d54..f7e2dc4cc 100644 --- a/openspec/changes/export-import-prompts-and-assistants/design.md +++ b/openspec/changes/export-import-prompts-and-assistants/design.md @@ -146,7 +146,7 @@ None. ### 5 - Which second workspace and second deployment should the tests use? > Context from the drafting agent, for question 5. -> The tests need a destination workspace and a destination deployment. The draft adds `LANGSTAR_TEST_DEST_WORKSPACE_ID`, a second workspace in the test organization so the existing API key works, and a second test deployment that the destination deployment fixture from [export-import-prompts-and-assistants Task 2.4 Add a destination deployment fixture](tasks.md#L15) creates from its own `langgraph.json` and its own name prefix, because the control plane allows one deployment per repository and graph config file and `find_reusable_deployment` matches by name prefix first. The control plane builds test deployments from `main`, so that `langgraph.json` merges to `main` in a pull request of its own, [export-import-prompts-and-assistants Task 2.3 Add the destination test graph's `langgraph.json`](tasks.md#L14), first. A second deployment costs build time and hosting in every CI run that creates it. Recommendation: a second workspace in the same organization, which codekiln creates, and one destination deployment that the destination deployment fixture from task 2.4 creates on the first run and the tests reuse after that. +> The import tests need a workspace and a deployment to import into. The draft has codekiln create a second workspace in the test organization, which the existing API key can reach, and the tests read its ID from `LANGSTAR_TEST_DEST_WORKSPACE_ID`. The destination deployment fixture from [export-import-prompts-and-assistants Task 2.4 Add a destination deployment fixture](tasks.md#L15) creates a second deployment of the test graph on the first CI run, and the tests reuse it after that. That deployment needs a `langgraph.json` of its own, because the control plane allows one deployment per repository and graph config file. The control plane builds test deployments from `main`, so a developer merges that file to `main` first, in [export-import-prompts-and-assistants Task 2.3 Add the destination test graph's `langgraph.json`](tasks.md#L14). A second deployment costs build time and hosting in every CI run that creates it. Recommendation: a second workspace in the same organization, which codekiln creates, and one destination deployment that the destination deployment fixture from task 2.4 creates on the first run and the tests reuse after that. @@ -174,6 +174,6 @@ None. ### 9 - Should the SDK field additions count as a breaking change? > Context from the drafting agent, for question 9. -> A new public field breaks only a crate that builds the struct with a struct literal. The only crate known to use the SDK is langstar's own CLI, and each pull request that adds a field updates the CLI's struct literals. `sdk/Cargo.toml` leaves the crate publishable, though, so an outside crate could break, and `docs/dev/git-scm-conventions.md` gives every API break a `BREAKING CHANGE:` footer, which makes the release a major version. The structs that gain fields are `Deployment`, `CommitManifestResponse` (which `PromptClient::get_commit` returns), `CommitData`, `CommitRequest`, `Assistant` and the assistant request structs. Marking them `#[non_exhaustive]` would stop outside crates from building them with struct literals at all, which breaks those crates once, in this release, and after that a new field would break nothing. [🚨 BREAKING CHANGE(sdk): move annotation-queue run calls to the platform items API · Pull Request #815](https://github.com/codekiln/langstar/pull/815) removes SDK functions and changes return types, marks itself breaking, and leaves that choice to codekiln too, so one answer to this question can set the policy for both pull requests. Recommendation: treat the additions as `feat` changes with no `BREAKING CHANGE:` footer, and name the new fields in the release notes. +> A new public field breaks only a crate that builds the struct with a struct literal. The only crate known to use the SDK is langstar's own CLI, and each pull request that adds a field updates the CLI's struct literals. `sdk/Cargo.toml` leaves the crate publishable, though, so an outside crate could break, and `docs/dev/git-scm-conventions.md` gives every API break a `BREAKING CHANGE:` footer, which makes the release a major version. The structs that gain fields are `Deployment`, `CommitManifestResponse` (which `PromptClient::get_commit` returns), `CommitData`, `CommitRequest`, `Assistant` and the assistant request structs. Marking them `#[non_exhaustive]` would stop every crate except the SDK itself from building them with struct literals, even with `..Default::default()`. langstar's own CLI and the SDK's integration tests build `Deployment`, `CommitRequest`, `CreateAssistantRequest` and `UpdateAssistantRequest` that way today, so the SDK would need a constructor for each struct before the CLI compiles again. After that one break, a new field would break nothing. [🚨 BREAKING CHANGE(sdk): move annotation-queue run calls to the platform items API · Pull Request #815](https://github.com/codekiln/langstar/pull/815) removes SDK functions and changes return types, marks itself breaking, and leaves that choice to codekiln too, so one answer to this question can set the policy for both pull requests. Recommendation: treat the additions as `feat` changes with no `BREAKING CHANGE:` footer, and name the new fields in the release notes. diff --git a/openspec/changes/export-import-prompts-and-assistants/tasks.md b/openspec/changes/export-import-prompts-and-assistants/tasks.md index 589f3f488..80a0861c8 100644 --- a/openspec/changes/export-import-prompts-and-assistants/tasks.md +++ b/openspec/changes/export-import-prompts-and-assistants/tasks.md @@ -1,6 +1,6 @@ # Tasks -Each numbered group below is one pull request, merged in the order listed. Task 2.1, which adds the destination workspace and the model provider key, waits on codekiln to create that workspace and an OpenAI key, so it gets a pull request of its own and the rest of group 2 goes ahead without it. Task 2.3, the destination test graph's `langgraph.json`, also gets a pull request of its own, because the control plane builds test deployments from `main` and task 2.4's fixture needs that file there. That pull request merges before group 6, prompt compare and verify, whose tests are the first to use the destination workspace. +Each numbered group below is one pull request, merged in the order listed. Task 2.3, which adds the destination test graph's `langgraph.json`, gets a pull request of its own that merges before the rest of group 2, because the control plane builds test deployments from `main` and task 2.4's fixture needs that file there. Task 2.1, which adds the destination workspace and the model provider key, also gets a pull request of its own, because it waits on codekiln to create that workspace and an OpenAI key. It merges before group 6, prompt compare and verify, whose tests are the first to use the destination workspace. ## 1. Check the API round trip before building on it From 9da89bd73044f20e68b32637e4d19a19ae4464b3 Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Tue, 13 Oct 2026 21:22:40 -0400 Subject: [PATCH 27/90] =?UTF-8?q?=F0=9F=93=9A=20docs(openspec):=20name=20t?= =?UTF-8?q?he=20destination=20deployment's=20integration-test=20prefix=20a?= =?UTF-8?q?nd=20its=20cleanup,=20list=20Assistant=20among=20the=20CLI's=20?= =?UTF-8?q?struct=20literals,=20and=20rename=20the=20workspace-id=20requir?= =?UTF-8?q?ement?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit #807 Export, import, compare, and verify prompts and assistants across LangSmith environments Co-Authored-By: Claude Opus 5.5 --- .../export-import-prompts-and-assistants/design.md | 8 +++++--- .../export-import-prompts-and-assistants/proposal.md | 4 ++-- .../specs/prompt/spec.md | 4 ++-- .../changes/export-import-prompts-and-assistants/tasks.md | 2 +- 4 files changed, 10 insertions(+), 8 deletions(-) diff --git a/openspec/changes/export-import-prompts-and-assistants/design.md b/openspec/changes/export-import-prompts-and-assistants/design.md index f7e2dc4cc..e30e22394 100644 --- a/openspec/changes/export-import-prompts-and-assistants/design.md +++ b/openspec/changes/export-import-prompts-and-assistants/design.md @@ -75,10 +75,12 @@ Compare reads the prompt or assistant from the destination, puts it in the shape ### Verify checks what the copy needs to run -Prompt verify checks that the workspace holds every secret the prompt's model reads. Export lists those names in `requires.secrets` by walking the manifest for LangChain secret references, `{"lc": 1, "type": "secret", "id": ["OPENAI_API_KEY"]}`, the shape the Python SDK's `_manifest_has_secrets` matches. Verify looks each name up among the workspace's secrets (`GET /api/v1/workspaces/current/secrets`, which `langstar secrets list` already calls) and prints the name of each missing secret and no secret value. Someone can edit the manifest in Git and forget `requires`, so import and verify walk the manifest again and stop with status 1, naming the field, when either list differs from the file's. +Prompt verify checks that the workspace holds every secret the prompt's model reads. Export lists those names in `requires.secrets` by walking the manifest for LangChain secret references, `{"lc": 1, "type": "secret", "id": ["OPENAI_API_KEY"]}`, the shape the Python SDK's `_manifest_has_secrets` matches. Verify looks each name up among the workspace's secrets (`GET /api/v1/workspaces/current/secrets`, which `langstar secrets list` already calls) and prints the name of each missing secret and no secret value. Prompt verify also notes when the workspace has no saved model configuration for the prompt's model, and leaves the exit status alone, because the prompt runs with the model its manifest carries. Export takes the model's class name, such as `ChatOpenAI`, from the last element of the chat model constructor's `id` in the manifest, such as `["langchain", "chat_models", "openai", "ChatOpenAI"]`, and stores it in `requires.model_provider`. Verify looks for a saved model configuration (`GET /api/v1/playground-settings`, which `langstar model-config list` calls) whose settings `id` ends in that class name. LangSmith also returns a `model_provider` field beside the manifest, which the file keeps as `commit.model_provider`; no SDK or spec shows what it holds, so in [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8) a developer records the value the live API returns. [#807 Export, import, compare, and verify prompts and assistants across LangSmith environments](https://github.com/codekiln/langstar/issues/807) asks for the note, and [Should prompt verify report a missing saved model configuration?](#7---should-prompt-verify-report-a-missing-saved-model-configuration) asks whether to keep it. +Someone can edit the manifest in Git and leave `requires` as it was, so import and verify read the secret names and the model class out of the manifest again, and stop with status 1 and name the field when either differs from `requires.secrets` or `requires.model_provider`. + Assistant verify checks the graph with `POST /assistants/search` filtered to the graph ID; every graph on a deployment has a default assistant, so an empty result means the deployment does not serve the graph. It checks each prompt in `requires.prompts`, such as `support-system:prod`, by asking LangSmith for the commit that tag points at (`GET /api/v1/commits/{owner}/{repo}/{tag}`, the request `langstar prompt pull --commit prod` already sends). A 404 means the workspace lacks the prompt or its tag. ### Import checks the destination's secrets, graph and prompts before it writes @@ -146,7 +148,7 @@ None. ### 5 - Which second workspace and second deployment should the tests use? > Context from the drafting agent, for question 5. -> The import tests need a workspace and a deployment to import into. The draft has codekiln create a second workspace in the test organization, which the existing API key can reach, and the tests read its ID from `LANGSTAR_TEST_DEST_WORKSPACE_ID`. The destination deployment fixture from [export-import-prompts-and-assistants Task 2.4 Add a destination deployment fixture](tasks.md#L15) creates a second deployment of the test graph on the first CI run, and the tests reuse it after that. That deployment needs a `langgraph.json` of its own, because the control plane allows one deployment per repository and graph config file. The control plane builds test deployments from `main`, so a developer merges that file to `main` first, in [export-import-prompts-and-assistants Task 2.3 Add the destination test graph's `langgraph.json`](tasks.md#L14). A second deployment costs build time and hosting in every CI run that creates it. Recommendation: a second workspace in the same organization, which codekiln creates, and one destination deployment that the destination deployment fixture from task 2.4 creates on the first run and the tests reuse after that. +> The import tests need a workspace and a deployment to import into. The draft has codekiln create a second workspace in the test organization, which the existing API key can reach, and the tests read its ID from `LANGSTAR_TEST_DEST_WORKSPACE_ID`. The destination deployment fixture from [export-import-prompts-and-assistants Task 2.4 Add a destination deployment fixture](tasks.md#L15) creates a second deployment of the test graph when no copy younger than four hours exists, because `.github/workflows/cleanup-test-deployments.yml` deletes every test deployment once it is four hours old. The first CI run after each cleanup therefore waits for two deployments to build instead of one. That deployment needs a `langgraph.json` of its own, because the control plane allows one deployment per repository and graph config file. The control plane builds test deployments from `main`, so a developer merges that file to `main` first, in [export-import-prompts-and-assistants Task 2.3 Add the destination test graph's `langgraph.json`](tasks.md#L14). Recommendation: a second workspace in the same organization, which codekiln creates, and one destination deployment that the destination deployment fixture from task 2.4 creates whenever the cleanup workflow has deleted it. @@ -174,6 +176,6 @@ None. ### 9 - Should the SDK field additions count as a breaking change? > Context from the drafting agent, for question 9. -> A new public field breaks only a crate that builds the struct with a struct literal. The only crate known to use the SDK is langstar's own CLI, and each pull request that adds a field updates the CLI's struct literals. `sdk/Cargo.toml` leaves the crate publishable, though, so an outside crate could break, and `docs/dev/git-scm-conventions.md` gives every API break a `BREAKING CHANGE:` footer, which makes the release a major version. The structs that gain fields are `Deployment`, `CommitManifestResponse` (which `PromptClient::get_commit` returns), `CommitData`, `CommitRequest`, `Assistant` and the assistant request structs. Marking them `#[non_exhaustive]` would stop every crate except the SDK itself from building them with struct literals, even with `..Default::default()`. langstar's own CLI and the SDK's integration tests build `Deployment`, `CommitRequest`, `CreateAssistantRequest` and `UpdateAssistantRequest` that way today, so the SDK would need a constructor for each struct before the CLI compiles again. After that one break, a new field would break nothing. [🚨 BREAKING CHANGE(sdk): move annotation-queue run calls to the platform items API · Pull Request #815](https://github.com/codekiln/langstar/pull/815) removes SDK functions and changes return types, marks itself breaking, and leaves that choice to codekiln too, so one answer to this question can set the policy for both pull requests. Recommendation: treat the additions as `feat` changes with no `BREAKING CHANGE:` footer, and name the new fields in the release notes. +> A new public field breaks only a crate that builds the struct with a struct literal. The only crate known to use the SDK is langstar's own CLI, and each pull request that adds a field updates the CLI's struct literals. `sdk/Cargo.toml` leaves the crate publishable, though, so an outside crate could break, and `docs/dev/git-scm-conventions.md` gives every API break a `BREAKING CHANGE:` footer, which makes the release a major version. The structs that gain fields are `Deployment`, `CommitManifestResponse` (which `PromptClient::get_commit` returns), `CommitData`, `CommitRequest`, `Assistant` and the assistant request structs. Marking them `#[non_exhaustive]` would stop every crate except the SDK itself from building them with struct literals, even with `..Default::default()`. langstar's own CLI builds `Assistant`, `CommitRequest`, `CreateAssistantRequest`, `UpdateAssistantRequest` and `Deployment` with struct literals today, and the SDK's integration tests build all of them except `Assistant` the same way, so the SDK would need a constructor for each of those five structs before the CLI and those tests compile again. After that one break, a new field would break nothing. [🚨 BREAKING CHANGE(sdk): move annotation-queue run calls to the platform items API · Pull Request #815](https://github.com/codekiln/langstar/pull/815) removes SDK functions and changes return types, marks itself breaking, and leaves that choice to codekiln too, so one answer to this question can set the policy for both pull requests. Recommendation: treat the additions as `feat` changes with no `BREAKING CHANGE:` footer, and name the new fields in the release notes. diff --git a/openspec/changes/export-import-prompts-and-assistants/proposal.md b/openspec/changes/export-import-prompts-and-assistants/proposal.md index f42b698af..e97833780 100644 --- a/openspec/changes/export-import-prompts-and-assistants/proposal.md +++ b/openspec/changes/export-import-prompts-and-assistants/proposal.md @@ -30,9 +30,9 @@ A prompt belongs to a workspace and an assistant belongs to a deployment, but bo ## Impact -- SDK: `sdk/src/prompts.rs` gains a commit read that asks for the model and returns it on `CommitManifestResponse`, a commit description on commit create, the new commit's `id`, a repo update, a commit list that returns each commit's `id`, and calls that list, create and move commit tags. `sdk/src/assistants.rs` gains the `context` and `version` fields, `graph_id` on update, and a search by graph ID. `sdk/src/deployments.rs` gains the deployment's `url`. Each new public field breaks a crate outside this repository that builds that SDK struct with a struct literal; design.md asks codekiln whether that makes the release a major version. +- SDK: `sdk/src/prompts.rs` gains a commit read that asks for the model and returns it on `CommitManifestResponse`, a commit description on commit create, the new commit's `id`, a repo update, a commit list that returns each commit's `id`, and calls that list, create and move commit tags. `sdk/src/assistants.rs` gains the `context` and `version` fields, `graph_id` on update, and a search by graph ID. `sdk/src/deployments.rs` gains the deployment's `url`. Each new public field breaks a crate outside this repository that builds that SDK struct with a struct literal; codekiln decides in [Should the SDK field additions count as a breaking change?](design.md#9---should-the-sdk-field-additions-count-as-a-breaking-change) whether that makes the release a major version. - CLI: new subcommands in `cli/src/commands/prompt.rs` and `cli/src/commands/assistant.rs`, and a shared module for the export file, the destination record, comparison and the exit status. `resolve_deployment_url` in `cli/src/deployment_utils.rs` also returns the Agent Server URL of a deployment built from GitHub, which every `langstar assistant` command needs. -- Tests need a second workspace, a second deployment and one model provider key for the structured output check. codekiln creates the workspace and the key, and langstar's tests read them through `fnox.toml` and the CI workflow; the destination deployment fixture from [export-import-prompts-and-assistants Task 2.4 Add a destination deployment fixture](tasks.md#L15) creates the deployment on the first run and the tests reuse it after that. +- Tests need a second workspace, a second deployment and one model provider key for the structured output check. codekiln creates the workspace and the key, and langstar's tests read them through `fnox.toml` and the CI workflow; the destination deployment fixture from [export-import-prompts-and-assistants Task 2.4 Add a destination deployment fixture](tasks.md#L15) creates the deployment whenever the cleanup workflow has deleted it, as the source fixture does today. - Docs: a new usage page on moving prompts and assistants between environments. ## Citations diff --git a/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md index e8edce179..86828f366 100644 --- a/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md +++ b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md @@ -45,7 +45,7 @@ Exporting the same commit twice SHALL produce identical files. The file SHALL ho ### Requirement: List the credentials a prompt needs without their values -The export file SHALL list, under `requires.secrets`, the name of every secret the manifest refers to, and under `requires.model_provider`, the class name at the end of the chat model constructor's `id` in the manifest, such as `ChatOpenAI`. It SHALL hold no secret values. Import and verify SHALL work out both lists again from the file's manifest, and when either differs from the file's `requires`, write nothing, name the field and exit with status 1. +The export file SHALL list, under `requires.secrets`, the name of every secret the manifest refers to, and under `requires.model_provider`, the class name that ends the chat model constructor's `id`, such as `ChatOpenAI`. It SHALL hold no secret values. Import and verify SHALL read the secret names and the model class out of the file's manifest again, and when either differs from `requires.secrets` or `requires.model_provider`, write nothing, name the field and exit with status 1. #### Scenario: Model reads an API key secret @@ -59,7 +59,7 @@ The export file SHALL list, under `requires.secrets`, the name of every secret t - **AND** a user runs `langstar prompt import my-prompt.json --workspace-id ` - **THEN** langstar creates no prompt, names `requires.secrets` and exits with status 1 -### Requirement: Import names its destination workspace +### Requirement: Import requires --workspace-id `langstar prompt import ` SHALL require `--workspace-id`. It SHALL refuse to run when the flag is missing, even when `LANGSMITH_WORKSPACE_ID` is set, and exit with status 1 and an error naming the flag. diff --git a/openspec/changes/export-import-prompts-and-assistants/tasks.md b/openspec/changes/export-import-prompts-and-assistants/tasks.md index 80a0861c8..0d7a64f08 100644 --- a/openspec/changes/export-import-prompts-and-assistants/tasks.md +++ b/openspec/changes/export-import-prompts-and-assistants/tasks.md @@ -12,7 +12,7 @@ Each numbered group below is one pull request, merged in the order listed. Task - [ ] 2.1 Add the destination workspace and the model provider key to `fnox.toml`, the CI workflow and `docs/dev/environment-variables.md`, with placeholders only; verify with `mise run check` and a CI run that reads both. This task waits on codekiln to create a second workspace in the test organization and an OpenAI API key, and to store both in 1Password and the repository's CI secrets and the key as the destination workspace's `OPENAI_API_KEY` secret. The import tests need the workspace as a place to import into, and its secret lets prompt import's secret check pass for a prompt with an OpenAI model. The key lets the check in 7.3 run each imported structured prompt once against a small OpenAI model, which bills that account on every CI run; it is the one test that shows a copy still returns structured data, where every other test compares text. - [ ] 2.2 Add `url` to `Deployment` in `sdk/src/deployments.rs`, with a method that returns `url` and falls back to `source_config.custom_url`, and make `resolve_deployment_url` in `cli/src/deployment_utils.rs` call that method, so that a deployment built from GitHub resolves; end the four `Deployment` struct literals in `sdk/tests/integration_deployment_workflow.rs` and `cli/src/commands/deployment.rs` with `..Default::default()`, because a crate that builds a `Deployment` with a struct literal stops compiling when the field is added; add a unit test for each case in `sdk/src/deployments.rs`, and verify by removing `#[ignore]` from `test_assistant_create_basic` in `cli/tests/assistant_command_test.rs` and running it against the test deployment - [ ] 2.3 Add the destination test graph's own `langgraph.json` under `tests/fixtures/`, naming the same test graph as `tests/fixtures/test-graph-deployment/langgraph.json`, in a pull request of its own that merges to `main` before the pull request for 2.4, because the control plane builds every test deployment from `main` (`TestDeploymentConfig::default` sets `branch` to `main`); verify by running `langstar deployment create` with `--branch` set to this pull request's branch and `--config-path` set to the new file, listing the test graph's assistants on that deployment, and deleting it -- [ ] 2.4 Add a destination deployment fixture to `sdk/src/test_utils.rs` and `cli/tests/common/fixtures.rs` with its own name prefix, one that does not start with `pr-integration-test`, and the `langgraph.json` from 2.3. `find_reusable_deployment` returns any deployment whose name starts with the fixture's prefix before it checks the graph config, and the control plane allows one deployment per repository and graph config file, so a shared prefix or a shared `langgraph.json` would hand back the source deployment, and a prefix starting with `pr-integration-test` would hand the destination deployment to the source fixture; verify with an integration test that asserts the destination and source deployment IDs differ and lists assistants on the destination deployment through the URL that 2.2 resolves +- [ ] 2.4 Add a destination deployment fixture to `sdk/src/test_utils.rs` and `cli/tests/common/fixtures.rs` that deploys the `langgraph.json` from 2.3 under the name prefix `dest-integration-test`. `find_reusable_deployment` returns any deployment whose name starts with the fixture's prefix before it checks the graph config, so this prefix keeps the source fixture, whose prefix is `pr-integration-test`, from picking up the destination deployment. The control plane allows one deployment per repository and graph config file, so the file from 2.3 keeps the destination fixture from picking up the source deployment. `.github/workflows/cleanup-test-deployments.yml` deletes every deployment whose name contains `integration-test` once it is four hours old, so the destination deployment is deleted and rebuilt on the same schedule as the source deployment. Verify with an integration test that asserts the destination and source deployment IDs differ and lists assistants on the destination deployment through the URL that 2.2 resolves ## 3. SDK reads and writes the fields export and import need From 2a02e0e04b9ef6d5fd3df1d345d7f3c29d0c0698 Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Tue, 13 Oct 2026 21:40:08 -0400 Subject: [PATCH 28/90] =?UTF-8?q?=F0=9F=93=9A=20docs(openspec):=20register?= =?UTF-8?q?=20the=20destination=20test=20graph=20under=20a=20second=20grap?= =?UTF-8?q?h=20ID=20for=20the=20assistant=20update=20tests,=20and=20list?= =?UTF-8?q?=20group=203's=20SDK=20structs=20by=20name?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit #807 Export, import, compare, and verify prompts and assistants across LangSmith environments Co-Authored-By: Claude Opus 5.5 --- .../export-import-prompts-and-assistants/design.md | 2 +- .../changes/export-import-prompts-and-assistants/tasks.md | 8 ++++---- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/openspec/changes/export-import-prompts-and-assistants/design.md b/openspec/changes/export-import-prompts-and-assistants/design.md index e30e22394..6764f7261 100644 --- a/openspec/changes/export-import-prompts-and-assistants/design.md +++ b/openspec/changes/export-import-prompts-and-assistants/design.md @@ -148,7 +148,7 @@ None. ### 5 - Which second workspace and second deployment should the tests use? > Context from the drafting agent, for question 5. -> The import tests need a workspace and a deployment to import into. The draft has codekiln create a second workspace in the test organization, which the existing API key can reach, and the tests read its ID from `LANGSTAR_TEST_DEST_WORKSPACE_ID`. The destination deployment fixture from [export-import-prompts-and-assistants Task 2.4 Add a destination deployment fixture](tasks.md#L15) creates a second deployment of the test graph when no copy younger than four hours exists, because `.github/workflows/cleanup-test-deployments.yml` deletes every test deployment once it is four hours old. The first CI run after each cleanup therefore waits for two deployments to build instead of one. That deployment needs a `langgraph.json` of its own, because the control plane allows one deployment per repository and graph config file. The control plane builds test deployments from `main`, so a developer merges that file to `main` first, in [export-import-prompts-and-assistants Task 2.3 Add the destination test graph's `langgraph.json`](tasks.md#L14). Recommendation: a second workspace in the same organization, which codekiln creates, and one destination deployment that the destination deployment fixture from task 2.4 creates whenever the cleanup workflow has deleted it. +> The import tests need a workspace and a deployment to import into. The draft has codekiln create a second workspace in the test organization, which the existing API key can reach, and the tests read its ID from `LANGSTAR_TEST_DEST_WORKSPACE_ID`. The destination deployment fixture from [export-import-prompts-and-assistants Task 2.4 Add a destination deployment fixture](tasks.md#L15) creates a second deployment of the test graph, registered under the graph IDs `test_graph` and `test_graph_alt` so that the update tests can move an assistant to a second graph, when no copy younger than four hours exists, because `.github/workflows/cleanup-test-deployments.yml` deletes every test deployment once it is four hours old. The first CI run after each cleanup therefore waits for two deployments to build instead of one. That deployment needs a `langgraph.json` of its own, because the control plane allows one deployment per repository and graph config file. The control plane builds test deployments from `main`, so a developer merges that file to `main` first, in [export-import-prompts-and-assistants Task 2.3 Add the destination test graph's `langgraph.json`](tasks.md#L14). Recommendation: a second workspace in the same organization, which codekiln creates, and one destination deployment that the destination deployment fixture from task 2.4 creates whenever the cleanup workflow has deleted it. diff --git a/openspec/changes/export-import-prompts-and-assistants/tasks.md b/openspec/changes/export-import-prompts-and-assistants/tasks.md index 0d7a64f08..5ad899fdb 100644 --- a/openspec/changes/export-import-prompts-and-assistants/tasks.md +++ b/openspec/changes/export-import-prompts-and-assistants/tasks.md @@ -11,16 +11,16 @@ Each numbered group below is one pull request, merged in the order listed. Task - [ ] 2.1 Add the destination workspace and the model provider key to `fnox.toml`, the CI workflow and `docs/dev/environment-variables.md`, with placeholders only; verify with `mise run check` and a CI run that reads both. This task waits on codekiln to create a second workspace in the test organization and an OpenAI API key, and to store both in 1Password and the repository's CI secrets and the key as the destination workspace's `OPENAI_API_KEY` secret. The import tests need the workspace as a place to import into, and its secret lets prompt import's secret check pass for a prompt with an OpenAI model. The key lets the check in 7.3 run each imported structured prompt once against a small OpenAI model, which bills that account on every CI run; it is the one test that shows a copy still returns structured data, where every other test compares text. - [ ] 2.2 Add `url` to `Deployment` in `sdk/src/deployments.rs`, with a method that returns `url` and falls back to `source_config.custom_url`, and make `resolve_deployment_url` in `cli/src/deployment_utils.rs` call that method, so that a deployment built from GitHub resolves; end the four `Deployment` struct literals in `sdk/tests/integration_deployment_workflow.rs` and `cli/src/commands/deployment.rs` with `..Default::default()`, because a crate that builds a `Deployment` with a struct literal stops compiling when the field is added; add a unit test for each case in `sdk/src/deployments.rs`, and verify by removing `#[ignore]` from `test_assistant_create_basic` in `cli/tests/assistant_command_test.rs` and running it against the test deployment -- [ ] 2.3 Add the destination test graph's own `langgraph.json` under `tests/fixtures/`, naming the same test graph as `tests/fixtures/test-graph-deployment/langgraph.json`, in a pull request of its own that merges to `main` before the pull request for 2.4, because the control plane builds every test deployment from `main` (`TestDeploymentConfig::default` sets `branch` to `main`); verify by running `langstar deployment create` with `--branch` set to this pull request's branch and `--config-path` set to the new file, listing the test graph's assistants on that deployment, and deleting it +- [ ] 2.3 Add the destination test graph's own `langgraph.json` under `tests/fixtures/`, registering the test graph from `tests/fixtures/test-graph-deployment/langgraph.json` under two graph IDs, `test_graph` and `test_graph_alt`, so that the update tests in 3.3 and 10.2 have a second graph ID to move an assistant to, in a pull request of its own that merges to `main` before the pull request for 2.4, because the control plane builds every test deployment from `main` (`TestDeploymentConfig::default` sets `branch` to `main`); verify by running `langstar deployment create` with `--branch` set to this pull request's branch and `--config-path` set to the new file, listing that deployment's assistants for each of `test_graph` and `test_graph_alt`, and deleting it - [ ] 2.4 Add a destination deployment fixture to `sdk/src/test_utils.rs` and `cli/tests/common/fixtures.rs` that deploys the `langgraph.json` from 2.3 under the name prefix `dest-integration-test`. `find_reusable_deployment` returns any deployment whose name starts with the fixture's prefix before it checks the graph config, so this prefix keeps the source fixture, whose prefix is `pr-integration-test`, from picking up the destination deployment. The control plane allows one deployment per repository and graph config file, so the file from 2.3 keeps the destination fixture from picking up the source deployment. `.github/workflows/cleanup-test-deployments.yml` deletes every deployment whose name contains `integration-test` once it is four hours old, so the destination deployment is deleted and rebuilt on the same schedule as the source deployment. Verify with an integration test that asserts the destination and source deployment IDs differ and lists assistants on the destination deployment through the URL that 2.2 resolves ## 3. SDK reads and writes the fields export and import need -Each task in this group adds a public field to an SDK struct (`CommitManifestResponse`, which `PromptClient::get_commit` returns, `CommitData`, `CommitRequest`, `Assistant` and the assistant request structs), which breaks a crate that builds that struct with a struct literal. Each task updates every struct literal of the struct it changes in this repository. Whether these pull requests and the one for task 2.2, which adds `url` to `Deployment`, carry a `BREAKING CHANGE:` footer waits on [Should the SDK field additions count as a breaking change?](design.md#9---should-the-sdk-field-additions-count-as-a-breaking-change). +Each task in this group adds a public field to one of these SDK structs: `CommitManifestResponse` (the struct `PromptClient::get_commit` returns), `CommitData`, `CommitRequest`, `Assistant`, `CreateAssistantRequest`, `UpdateAssistantRequest` and `AssistantSearchRequest`. A crate that builds one of them with a struct literal stops compiling, so each task updates every struct literal of its struct in this repository. Whether this group's pull request and task 2.2's, which adds `url` to `Deployment`, carry a `BREAKING CHANGE:` footer waits on [Should the SDK field additions count as a breaking change?](design.md#9---should-the-sdk-field-additions-count-as-a-breaking-change). - [ ] 3.1 Add `description`, `model_config` and `model_provider` to `CommitManifestResponse`, and a commit read that sends `include=model` and returns that struct, to `sdk/src/prompts.rs`; verify with an httpmock test that asserts the query string and the parsed fields, and an integration test that reads a prompt with a model - [ ] 3.2 Add the commit's `id` to `CommitData` in `sdk/src/prompts.rs`, a call that lists a prompt's commits with their `id` and hash (`GET /api/v1/commits/{owner}/{repo}`), and calls that list, create and move commit tags, which name their commit by that `id` in `POST /api/v1/repos/{owner}/{repo}/tags` and `PATCH /api/v1/repos/{owner}/{repo}/tags/{tag_name}`; verify with httpmock tests that assert `commit_id` in each request body, and an integration test that pushes two commits, tags the first, finds the second commit's `id` through the commit list, moves the tag to it, and reads the tag back through the SDK -- [ ] 3.3 Add `context` and `version` to `Assistant`, `context` and `graph_id` to `UpdateAssistantRequest`, `context` to `CreateAssistantRequest`, and a `graph_id` filter to `AssistantSearchRequest`, in `sdk/src/assistants.rs`; verify with httpmock tests of the request bodies and an integration test that creates an assistant with context, reads it back with its version, finds it by graph ID, and updates its graph ID and reads the new graph ID back +- [ ] 3.3 Add `context` and `version` to `Assistant`, `context` and `graph_id` to `UpdateAssistantRequest`, `context` to `CreateAssistantRequest`, and a `graph_id` filter to `AssistantSearchRequest`, in `sdk/src/assistants.rs`; verify with httpmock tests of the request bodies and an integration test that creates an assistant with context on the destination deployment from 2.4, reads it back with its version, finds it by graph ID, moves it from `test_graph` to `test_graph_alt`, and reads `test_graph_alt` back. Group 2 merges before this group, so the destination deployment fixture from 2.4 exists by then; the source deployment serves only `test_graph`, so the test cannot run there - [ ] 3.4 Add `description` to `CommitRequest`, and a repo update that sends `PATCH /api/v1/repos/{owner}/{repo}` with `description`, `readme`, `tags` and `is_public`, to `sdk/src/prompts.rs`; verify with httpmock tests of the request bodies and an integration test that pushes a commit with a description, updates the repo's description, readme, tags and visibility, and reads each back ## 4. Shared code for the prompt and assistant commands: the export file, the comparison and the report @@ -62,7 +62,7 @@ Each task in this group adds a public field to an SDK struct (`CommitManifestRes ## 10. assistant import - [ ] 10.1 Add `langstar assistant import` with `--workspace-id`, the graph check and prompt check of `assistant verify` from 9.2 run before it writes, creation, read-back, the destination record and `--record`; verify with an integration test that creates a tagged prompt in the test workspace, imports an assistant that refers to it into the destination deployment and checks name, graph ID, configuration, context, metadata and prompt references through the Agent Server API, that `--record ` writes the destination record to that path, and that a file whose graph ID the deployment lacks and a file with a missing prompt tag each create no assistant and exit with status 3, and an httpmock test in which the assistant read back differs from the file, asserting that import exits with status 3, that the destination record holds the new assistant ID, and that a second import creates no assistant, and a CLI test that `assistant import` of a prompt export exits with status 1, names `kind` and the value `prompt`, and creates no assistant -- [ ] 10.2 Add `--update`, `--assistant-id` and `--dry-run`; verify with integration tests that a repeated import creates no assistant, that `--update` changes the recorded one, its graph ID among the fields, that `--update --assistant-id ` naming an assistant other than the recorded one updates that assistant, leaves the recorded one unchanged and writes the new ID to the destination record, that `--update` of an assistant with metadata `{"kept": 1, "stale": 2}` from a file with `{"kept": 1}` leaves `{"kept": 1}` or reports `/metadata/stale` and exits with status 3, whichever task 1.2 found the server does, that `--update` of an assistant with a description and context from a file with neither clears both or reports `/description` and `/context` and exits with status 3, whichever task 1.2 found the server does, that a deleted recorded assistant exits with status 1, and that `--dry-run` creates no assistant, changes none and leaves the destination record as it was +- [ ] 10.2 Add `--update`, `--assistant-id` and `--dry-run`; verify with integration tests that a repeated import creates no assistant, that `--update` changes the recorded one, moving it from `test_graph` to `test_graph_alt` among the fields, that `--update --assistant-id ` naming an assistant other than the recorded one updates that assistant, leaves the recorded one unchanged and writes the new ID to the destination record, that `--update` of an assistant with metadata `{"kept": 1, "stale": 2}` from a file with `{"kept": 1}` leaves `{"kept": 1}` or reports `/metadata/stale` and exits with status 3, whichever task 1.2 found the server does, that `--update` of an assistant with a description and context from a file with neither clears both or reports `/description` and `/context` and exits with status 3, whichever task 1.2 found the server does, that a deleted recorded assistant exits with status 1, and that `--dry-run` creates no assistant, changes none and leaves the destination record as it was - [ ] 10.3 Add an end-to-end test that imports prompts into the destination workspace, then imports an assistant that refers to them onto the destination deployment and runs `assistant verify`, passing `--workspace-id` set to the destination workspace to both assistant commands; verify that it passes, and that verify fails after deleting one prompt tag in the destination workspace - [ ] 10.4 Document assistant import, the deployment scope of assistants, and how export and import relate to the `create`, `get` and `update` commands of [#728 Implement CRUD command structure (create, get, update) for prompt commands](https://github.com/codekiln/langstar/issues/728); verify that each documented command runs as written From b65c1f4465484bd72fa2055ce46b0dffe782cc41 Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Tue, 13 Oct 2026 21:57:31 -0400 Subject: [PATCH 29/90] =?UTF-8?q?=F0=9F=93=9A=20docs(openspec):=20merge=20?= =?UTF-8?q?task=202.2=20before=20task=202.3=20so=20its=20check=20lists=20a?= =?UTF-8?q?ssistants=20through=20langstar,=20and=20cite=20find=5Freusable?= =?UTF-8?q?=5Fdeployment=20and=20the=20409=20for=20the=20destination=20dep?= =?UTF-8?q?loyment?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit #807 Export, import, compare, and verify prompts and assistants across LangSmith environments Co-Authored-By: Claude Opus 5.5 --- .../changes/export-import-prompts-and-assistants/design.md | 2 +- .../changes/export-import-prompts-and-assistants/tasks.md | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/openspec/changes/export-import-prompts-and-assistants/design.md b/openspec/changes/export-import-prompts-and-assistants/design.md index 6764f7261..ad07a237e 100644 --- a/openspec/changes/export-import-prompts-and-assistants/design.md +++ b/openspec/changes/export-import-prompts-and-assistants/design.md @@ -148,7 +148,7 @@ None. ### 5 - Which second workspace and second deployment should the tests use? > Context from the drafting agent, for question 5. -> The import tests need a workspace and a deployment to import into. The draft has codekiln create a second workspace in the test organization, which the existing API key can reach, and the tests read its ID from `LANGSTAR_TEST_DEST_WORKSPACE_ID`. The destination deployment fixture from [export-import-prompts-and-assistants Task 2.4 Add a destination deployment fixture](tasks.md#L15) creates a second deployment of the test graph, registered under the graph IDs `test_graph` and `test_graph_alt` so that the update tests can move an assistant to a second graph, when no copy younger than four hours exists, because `.github/workflows/cleanup-test-deployments.yml` deletes every test deployment once it is four hours old. The first CI run after each cleanup therefore waits for two deployments to build instead of one. That deployment needs a `langgraph.json` of its own, because the control plane allows one deployment per repository and graph config file. The control plane builds test deployments from `main`, so a developer merges that file to `main` first, in [export-import-prompts-and-assistants Task 2.3 Add the destination test graph's `langgraph.json`](tasks.md#L14). Recommendation: a second workspace in the same organization, which codekiln creates, and one destination deployment that the destination deployment fixture from task 2.4 creates whenever the cleanup workflow has deleted it. +> The import tests need a workspace and a deployment to import into. The draft has codekiln create a second workspace in the test organization, which the existing API key can reach, and the tests read its ID from `LANGSTAR_TEST_DEST_WORKSPACE_ID`. The destination deployment fixture from [export-import-prompts-and-assistants Task 2.4 Add a destination deployment fixture](tasks.md#L15) deploys the test graph a second time, under the graph IDs `test_graph` and `test_graph_alt`, so that the update tests can move an assistant from one graph to the other. `.github/workflows/cleanup-test-deployments.yml` runs every four hours and deletes each test deployment that is four hours old or older, and the fixture builds a new destination deployment on the next CI run. The first CI run after each cleanup therefore waits for two deployments to build instead of one. That deployment needs a `langgraph.json` of its own, because the control plane refuses a second deployment of the same agent environment, and the verification in [export-import-prompts-and-assistants Task 2.3 Add the destination test graph's `langgraph.json`](tasks.md#L14) checks that a second graph config file makes a new one. The control plane builds test deployments from `main`, so a developer merges that file to `main` before the fixture's pull request, in [export-import-prompts-and-assistants Task 2.3 Add the destination test graph's `langgraph.json`](tasks.md#L14). Recommendation: a second workspace in the same organization, which codekiln creates, and one destination deployment that the destination deployment fixture from task 2.4 creates whenever the cleanup workflow has deleted it. diff --git a/openspec/changes/export-import-prompts-and-assistants/tasks.md b/openspec/changes/export-import-prompts-and-assistants/tasks.md index 5ad899fdb..c287c30e1 100644 --- a/openspec/changes/export-import-prompts-and-assistants/tasks.md +++ b/openspec/changes/export-import-prompts-and-assistants/tasks.md @@ -1,6 +1,6 @@ # Tasks -Each numbered group below is one pull request, merged in the order listed. Task 2.3, which adds the destination test graph's `langgraph.json`, gets a pull request of its own that merges before the rest of group 2, because the control plane builds test deployments from `main` and task 2.4's fixture needs that file there. Task 2.1, which adds the destination workspace and the model provider key, also gets a pull request of its own, because it waits on codekiln to create that workspace and an OpenAI key. It merges before group 6, prompt compare and verify, whose tests are the first to use the destination workspace. +Each numbered group below is one pull request, merged in the order listed. Group 2 is the exception: tasks 2.2, 2.3 and 2.4 each get a pull request of their own, merged in that order. Task 2.3's check lists assistants through the deployment URL that 2.2 resolves, and the control plane builds test deployments from `main`, so task 2.4's fixture needs 2.3's `langgraph.json` there. Task 2.1, which adds the destination workspace and the model provider key, also gets a pull request of its own, because it waits on codekiln to create that workspace and an OpenAI key. It merges before group 6, prompt compare and verify, whose tests are the first to use the destination workspace. ## 1. Check the API round trip before building on it @@ -11,8 +11,8 @@ Each numbered group below is one pull request, merged in the order listed. Task - [ ] 2.1 Add the destination workspace and the model provider key to `fnox.toml`, the CI workflow and `docs/dev/environment-variables.md`, with placeholders only; verify with `mise run check` and a CI run that reads both. This task waits on codekiln to create a second workspace in the test organization and an OpenAI API key, and to store both in 1Password and the repository's CI secrets and the key as the destination workspace's `OPENAI_API_KEY` secret. The import tests need the workspace as a place to import into, and its secret lets prompt import's secret check pass for a prompt with an OpenAI model. The key lets the check in 7.3 run each imported structured prompt once against a small OpenAI model, which bills that account on every CI run; it is the one test that shows a copy still returns structured data, where every other test compares text. - [ ] 2.2 Add `url` to `Deployment` in `sdk/src/deployments.rs`, with a method that returns `url` and falls back to `source_config.custom_url`, and make `resolve_deployment_url` in `cli/src/deployment_utils.rs` call that method, so that a deployment built from GitHub resolves; end the four `Deployment` struct literals in `sdk/tests/integration_deployment_workflow.rs` and `cli/src/commands/deployment.rs` with `..Default::default()`, because a crate that builds a `Deployment` with a struct literal stops compiling when the field is added; add a unit test for each case in `sdk/src/deployments.rs`, and verify by removing `#[ignore]` from `test_assistant_create_basic` in `cli/tests/assistant_command_test.rs` and running it against the test deployment -- [ ] 2.3 Add the destination test graph's own `langgraph.json` under `tests/fixtures/`, registering the test graph from `tests/fixtures/test-graph-deployment/langgraph.json` under two graph IDs, `test_graph` and `test_graph_alt`, so that the update tests in 3.3 and 10.2 have a second graph ID to move an assistant to, in a pull request of its own that merges to `main` before the pull request for 2.4, because the control plane builds every test deployment from `main` (`TestDeploymentConfig::default` sets `branch` to `main`); verify by running `langstar deployment create` with `--branch` set to this pull request's branch and `--config-path` set to the new file, listing that deployment's assistants for each of `test_graph` and `test_graph_alt`, and deleting it -- [ ] 2.4 Add a destination deployment fixture to `sdk/src/test_utils.rs` and `cli/tests/common/fixtures.rs` that deploys the `langgraph.json` from 2.3 under the name prefix `dest-integration-test`. `find_reusable_deployment` returns any deployment whose name starts with the fixture's prefix before it checks the graph config, so this prefix keeps the source fixture, whose prefix is `pr-integration-test`, from picking up the destination deployment. The control plane allows one deployment per repository and graph config file, so the file from 2.3 keeps the destination fixture from picking up the source deployment. `.github/workflows/cleanup-test-deployments.yml` deletes every deployment whose name contains `integration-test` once it is four hours old, so the destination deployment is deleted and rebuilt on the same schedule as the source deployment. Verify with an integration test that asserts the destination and source deployment IDs differ and lists assistants on the destination deployment through the URL that 2.2 resolves +- [ ] 2.3 Add the destination test graph's own `langgraph.json` under `tests/fixtures/`, registering the test graph from `tests/fixtures/test-graph-deployment/langgraph.json` under two graph IDs, `test_graph` and `test_graph_alt`, so that the update tests in 3.3 and 10.2 have a second graph ID to move an assistant to, in a pull request of its own that merges to `main` after the pull request for 2.2 and before the pull request for 2.4, because the control plane builds every test deployment from `main` (`TestDeploymentConfig::default` sets `branch` to `main`); verify by running `langstar deployment create` while the source test deployment is live, with `--branch` set to this pull request's branch and `--config-path` set to the new file, then running `langstar assistant list --deployment `, which reaches a deployment built from GitHub through the `url` fallback from 2.2, and finding an assistant with graph ID `test_graph` and one with `test_graph_alt`; then delete the deployment +- [ ] 2.4 Add a destination deployment fixture to `sdk/src/test_utils.rs` and `cli/tests/common/fixtures.rs` that deploys the `langgraph.json` from 2.3 under the name prefix `dest-integration-test`. `find_reusable_deployment` returns any deployment whose name starts with the fixture's prefix before it checks the graph config, so this prefix keeps the source fixture, whose prefix is `pr-integration-test`, from picking up the destination deployment. `find_reusable_deployment` then falls back to any live deployment built from the same repository and graph config file, so the separate file from 2.3 keeps the destination fixture from picking up the source deployment. The control plane answers a second deployment of one agent environment with 409 "A deployment already exists for this agent environment". No document says which fields make an agent environment, and the verification in 2.3 shows whether a second graph config file in the same repository makes a new one. `.github/workflows/cleanup-test-deployments.yml` runs every four hours and deletes each deployment whose name contains `integration-test` once it is four hours old or older, so the destination deployment is deleted and rebuilt on the same schedule as the source deployment. Verify with an integration test that asserts the destination and source deployment IDs differ and lists assistants on the destination deployment through the URL that 2.2 resolves ## 3. SDK reads and writes the fields export and import need From f67994d37006c61b80d9064deaad070f5b11d28e Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Tue, 13 Oct 2026 22:10:10 -0400 Subject: [PATCH 30/90] =?UTF-8?q?=F0=9F=93=9A=20docs(openspec):=20state=20?= =?UTF-8?q?the=20test-environment=20merge=20order=20once=20in=20the=20task?= =?UTF-8?q?s=20preamble,=20and=20trim=20it=20from=20task=202.3,=20task=203?= =?UTF-8?q?.3=20and=20question=205?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit #807 Export, import, compare, and verify prompts and assistants across LangSmith environments Co-Authored-By: Claude Opus 5.5 --- .../changes/export-import-prompts-and-assistants/design.md | 2 +- .../changes/export-import-prompts-and-assistants/tasks.md | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/openspec/changes/export-import-prompts-and-assistants/design.md b/openspec/changes/export-import-prompts-and-assistants/design.md index ad07a237e..751fbeac9 100644 --- a/openspec/changes/export-import-prompts-and-assistants/design.md +++ b/openspec/changes/export-import-prompts-and-assistants/design.md @@ -148,7 +148,7 @@ None. ### 5 - Which second workspace and second deployment should the tests use? > Context from the drafting agent, for question 5. -> The import tests need a workspace and a deployment to import into. The draft has codekiln create a second workspace in the test organization, which the existing API key can reach, and the tests read its ID from `LANGSTAR_TEST_DEST_WORKSPACE_ID`. The destination deployment fixture from [export-import-prompts-and-assistants Task 2.4 Add a destination deployment fixture](tasks.md#L15) deploys the test graph a second time, under the graph IDs `test_graph` and `test_graph_alt`, so that the update tests can move an assistant from one graph to the other. `.github/workflows/cleanup-test-deployments.yml` runs every four hours and deletes each test deployment that is four hours old or older, and the fixture builds a new destination deployment on the next CI run. The first CI run after each cleanup therefore waits for two deployments to build instead of one. That deployment needs a `langgraph.json` of its own, because the control plane refuses a second deployment of the same agent environment, and the verification in [export-import-prompts-and-assistants Task 2.3 Add the destination test graph's `langgraph.json`](tasks.md#L14) checks that a second graph config file makes a new one. The control plane builds test deployments from `main`, so a developer merges that file to `main` before the fixture's pull request, in [export-import-prompts-and-assistants Task 2.3 Add the destination test graph's `langgraph.json`](tasks.md#L14). Recommendation: a second workspace in the same organization, which codekiln creates, and one destination deployment that the destination deployment fixture from task 2.4 creates whenever the cleanup workflow has deleted it. +> The import tests need a workspace and a deployment to import into. The draft has codekiln create a second workspace in the test organization, which the existing API key can reach, and the tests read its ID from `LANGSTAR_TEST_DEST_WORKSPACE_ID`. The destination deployment fixture from [export-import-prompts-and-assistants Task 2.4 Add a destination deployment fixture](tasks.md#L15) deploys the test graph a second time from a `langgraph.json` of its own, which registers the graph as both `test_graph` and `test_graph_alt` so the update tests can move an assistant from one graph to the other. `.github/workflows/cleanup-test-deployments.yml` deletes each test deployment once it is four hours old and the fixture rebuilds it on the next CI run, so the first CI run after each cleanup waits for two deployments to build instead of one. Recommendation: a second workspace in the same organization, which codekiln creates, and one destination deployment that the fixture from task 2.4 rebuilds whenever the cleanup workflow has deleted it. diff --git a/openspec/changes/export-import-prompts-and-assistants/tasks.md b/openspec/changes/export-import-prompts-and-assistants/tasks.md index c287c30e1..56d5194ac 100644 --- a/openspec/changes/export-import-prompts-and-assistants/tasks.md +++ b/openspec/changes/export-import-prompts-and-assistants/tasks.md @@ -1,6 +1,6 @@ # Tasks -Each numbered group below is one pull request, merged in the order listed. Group 2 is the exception: tasks 2.2, 2.3 and 2.4 each get a pull request of their own, merged in that order. Task 2.3's check lists assistants through the deployment URL that 2.2 resolves, and the control plane builds test deployments from `main`, so task 2.4's fixture needs 2.3's `langgraph.json` there. Task 2.1, which adds the destination workspace and the model provider key, also gets a pull request of its own, because it waits on codekiln to create that workspace and an OpenAI key. It merges before group 6, prompt compare and verify, whose tests are the first to use the destination workspace. +Each numbered group below is one pull request, merged in the order listed, except group 2, whose tasks each get a pull request of their own. Tasks 2.2, 2.3 and 2.4 merge in that order: the check in 2.3 lists assistants through the deployment URL that 2.2 resolves, and the fixture in 2.4 deploys from `main`, so the `langgraph.json` from 2.3 has to be on `main` first. Task 2.1 waits on codekiln to create the destination workspace and an OpenAI key, and can merge any time before group 6, whose prompt compare and verify tests are the first to use that workspace. ## 1. Check the API round trip before building on it @@ -11,7 +11,7 @@ Each numbered group below is one pull request, merged in the order listed. Group - [ ] 2.1 Add the destination workspace and the model provider key to `fnox.toml`, the CI workflow and `docs/dev/environment-variables.md`, with placeholders only; verify with `mise run check` and a CI run that reads both. This task waits on codekiln to create a second workspace in the test organization and an OpenAI API key, and to store both in 1Password and the repository's CI secrets and the key as the destination workspace's `OPENAI_API_KEY` secret. The import tests need the workspace as a place to import into, and its secret lets prompt import's secret check pass for a prompt with an OpenAI model. The key lets the check in 7.3 run each imported structured prompt once against a small OpenAI model, which bills that account on every CI run; it is the one test that shows a copy still returns structured data, where every other test compares text. - [ ] 2.2 Add `url` to `Deployment` in `sdk/src/deployments.rs`, with a method that returns `url` and falls back to `source_config.custom_url`, and make `resolve_deployment_url` in `cli/src/deployment_utils.rs` call that method, so that a deployment built from GitHub resolves; end the four `Deployment` struct literals in `sdk/tests/integration_deployment_workflow.rs` and `cli/src/commands/deployment.rs` with `..Default::default()`, because a crate that builds a `Deployment` with a struct literal stops compiling when the field is added; add a unit test for each case in `sdk/src/deployments.rs`, and verify by removing `#[ignore]` from `test_assistant_create_basic` in `cli/tests/assistant_command_test.rs` and running it against the test deployment -- [ ] 2.3 Add the destination test graph's own `langgraph.json` under `tests/fixtures/`, registering the test graph from `tests/fixtures/test-graph-deployment/langgraph.json` under two graph IDs, `test_graph` and `test_graph_alt`, so that the update tests in 3.3 and 10.2 have a second graph ID to move an assistant to, in a pull request of its own that merges to `main` after the pull request for 2.2 and before the pull request for 2.4, because the control plane builds every test deployment from `main` (`TestDeploymentConfig::default` sets `branch` to `main`); verify by running `langstar deployment create` while the source test deployment is live, with `--branch` set to this pull request's branch and `--config-path` set to the new file, then running `langstar assistant list --deployment `, which reaches a deployment built from GitHub through the `url` fallback from 2.2, and finding an assistant with graph ID `test_graph` and one with `test_graph_alt`; then delete the deployment +- [ ] 2.3 Add a second `langgraph.json` under `tests/fixtures/` that registers the test graph from `tests/fixtures/test-graph-deployment/langgraph.json` under two graph IDs, `test_graph` and `test_graph_alt`, so the update tests in 3.3 and 10.2 can move an assistant from one graph to the other. To verify, run `langstar deployment create` with `--branch` set to this pull request's branch and `--config-path` set to the new file while the source test deployment is live, run `langstar assistant list --deployment `, find one assistant with graph ID `test_graph` and one with `test_graph_alt`, then delete the deployment. - [ ] 2.4 Add a destination deployment fixture to `sdk/src/test_utils.rs` and `cli/tests/common/fixtures.rs` that deploys the `langgraph.json` from 2.3 under the name prefix `dest-integration-test`. `find_reusable_deployment` returns any deployment whose name starts with the fixture's prefix before it checks the graph config, so this prefix keeps the source fixture, whose prefix is `pr-integration-test`, from picking up the destination deployment. `find_reusable_deployment` then falls back to any live deployment built from the same repository and graph config file, so the separate file from 2.3 keeps the destination fixture from picking up the source deployment. The control plane answers a second deployment of one agent environment with 409 "A deployment already exists for this agent environment". No document says which fields make an agent environment, and the verification in 2.3 shows whether a second graph config file in the same repository makes a new one. `.github/workflows/cleanup-test-deployments.yml` runs every four hours and deletes each deployment whose name contains `integration-test` once it is four hours old or older, so the destination deployment is deleted and rebuilt on the same schedule as the source deployment. Verify with an integration test that asserts the destination and source deployment IDs differ and lists assistants on the destination deployment through the URL that 2.2 resolves ## 3. SDK reads and writes the fields export and import need @@ -20,7 +20,7 @@ Each task in this group adds a public field to one of these SDK structs: `Commit - [ ] 3.1 Add `description`, `model_config` and `model_provider` to `CommitManifestResponse`, and a commit read that sends `include=model` and returns that struct, to `sdk/src/prompts.rs`; verify with an httpmock test that asserts the query string and the parsed fields, and an integration test that reads a prompt with a model - [ ] 3.2 Add the commit's `id` to `CommitData` in `sdk/src/prompts.rs`, a call that lists a prompt's commits with their `id` and hash (`GET /api/v1/commits/{owner}/{repo}`), and calls that list, create and move commit tags, which name their commit by that `id` in `POST /api/v1/repos/{owner}/{repo}/tags` and `PATCH /api/v1/repos/{owner}/{repo}/tags/{tag_name}`; verify with httpmock tests that assert `commit_id` in each request body, and an integration test that pushes two commits, tags the first, finds the second commit's `id` through the commit list, moves the tag to it, and reads the tag back through the SDK -- [ ] 3.3 Add `context` and `version` to `Assistant`, `context` and `graph_id` to `UpdateAssistantRequest`, `context` to `CreateAssistantRequest`, and a `graph_id` filter to `AssistantSearchRequest`, in `sdk/src/assistants.rs`; verify with httpmock tests of the request bodies and an integration test that creates an assistant with context on the destination deployment from 2.4, reads it back with its version, finds it by graph ID, moves it from `test_graph` to `test_graph_alt`, and reads `test_graph_alt` back. Group 2 merges before this group, so the destination deployment fixture from 2.4 exists by then; the source deployment serves only `test_graph`, so the test cannot run there +- [ ] 3.3 Add `context` and `version` to `Assistant`, `context` and `graph_id` to `UpdateAssistantRequest`, `context` to `CreateAssistantRequest`, and a `graph_id` filter to `AssistantSearchRequest`, in `sdk/src/assistants.rs`; verify with httpmock tests of the request bodies and an integration test that creates an assistant with context on the destination deployment from 2.4, reads it back with its version, finds it by graph ID, moves it from `test_graph` to `test_graph_alt`, and reads `test_graph_alt` back. The test runs on the destination deployment because the source deployment serves only `test_graph`, and task 2.4, which adds the destination fixture, merges before this group. - [ ] 3.4 Add `description` to `CommitRequest`, and a repo update that sends `PATCH /api/v1/repos/{owner}/{repo}` with `description`, `readme`, `tags` and `is_public`, to `sdk/src/prompts.rs`; verify with httpmock tests of the request bodies and an integration test that pushes a commit with a description, updates the repo's description, readme, tags and visibility, and reads each back ## 4. Shared code for the prompt and assistant commands: the export file, the comparison and the report From e4ae746d51f1f3f9ccf3fc97787148c3b4df2bce Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Tue, 13 Oct 2026 22:28:01 -0400 Subject: [PATCH 31/90] =?UTF-8?q?=F0=9F=93=9A=20docs(openspec):=20name=20t?= =?UTF-8?q?he=20409=20risk=20behind=20the=20destination=20deployment,=20te?= =?UTF-8?q?st=20--prompt-ref=20export,=20and=20handle=20prompts=20without?= =?UTF-8?q?=20a=20model?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Design question 5 and Risks / Trade-offs now say the control plane may refuse a second deployment from this repository with 409, and that task 2.3 finds out and sends question 5 back to codekiln. Task 8.1 tests that export writes the --prompt-ref pointer and prompt name. A prompt without a model exports requires.model_provider as null, and import and verify skip the saved model configuration check for it. #807 Export, import, compare, and verify prompts and assistants across LangSmith environments Co-Authored-By: Claude Opus 5.5 --- .../export-import-prompts-and-assistants/design.md | 7 ++++--- .../proposal.md | 2 +- .../specs/prompt/spec.md | 13 ++++++++++++- .../export-import-prompts-and-assistants/tasks.md | 10 +++++----- 4 files changed, 22 insertions(+), 10 deletions(-) diff --git a/openspec/changes/export-import-prompts-and-assistants/design.md b/openspec/changes/export-import-prompts-and-assistants/design.md index 751fbeac9..ab4f3d65b 100644 --- a/openspec/changes/export-import-prompts-and-assistants/design.md +++ b/openspec/changes/export-import-prompts-and-assistants/design.md @@ -77,9 +77,9 @@ Compare reads the prompt or assistant from the destination, puts it in the shape Prompt verify checks that the workspace holds every secret the prompt's model reads. Export lists those names in `requires.secrets` by walking the manifest for LangChain secret references, `{"lc": 1, "type": "secret", "id": ["OPENAI_API_KEY"]}`, the shape the Python SDK's `_manifest_has_secrets` matches. Verify looks each name up among the workspace's secrets (`GET /api/v1/workspaces/current/secrets`, which `langstar secrets list` already calls) and prints the name of each missing secret and no secret value. -Prompt verify also notes when the workspace has no saved model configuration for the prompt's model, and leaves the exit status alone, because the prompt runs with the model its manifest carries. Export takes the model's class name, such as `ChatOpenAI`, from the last element of the chat model constructor's `id` in the manifest, such as `["langchain", "chat_models", "openai", "ChatOpenAI"]`, and stores it in `requires.model_provider`. Verify looks for a saved model configuration (`GET /api/v1/playground-settings`, which `langstar model-config list` calls) whose settings `id` ends in that class name. LangSmith also returns a `model_provider` field beside the manifest, which the file keeps as `commit.model_provider`; no SDK or spec shows what it holds, so in [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8) a developer records the value the live API returns. [#807 Export, import, compare, and verify prompts and assistants across LangSmith environments](https://github.com/codekiln/langstar/issues/807) asks for the note, and [Should prompt verify report a missing saved model configuration?](#7---should-prompt-verify-report-a-missing-saved-model-configuration) asks whether to keep it. +Prompt verify also notes when the workspace has no saved model configuration for the prompt's model, and leaves the exit status alone, because the prompt runs with the model its manifest carries. Export takes the model's class name, such as `ChatOpenAI`, from the last element of the chat model constructor's `id` in the manifest, such as `["langchain", "chat_models", "openai", "ChatOpenAI"]`, and stores it in `requires.model_provider`. Verify looks for a saved model configuration (`GET /api/v1/playground-settings`, which `langstar model-config list` calls) whose settings `id` ends in that class name. LangSmith also returns a `model_provider` field beside the manifest, which the file keeps as `commit.model_provider`; no SDK or spec shows what it holds, so in [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8) a developer records the value the live API returns. [#807 Export, import, compare, and verify prompts and assistants across LangSmith environments](https://github.com/codekiln/langstar/issues/807) asks for the note, and [Should prompt verify report a missing saved model configuration?](#7---should-prompt-verify-report-a-missing-saved-model-configuration) asks whether to keep it. A prompt saved without a model has no chat model constructor in its manifest, so export writes `requires.model_provider` as null, and verify and import skip the saved model configuration check for it and still run the secret check. -Someone can edit the manifest in Git and leave `requires` as it was, so import and verify read the secret names and the model class out of the manifest again, and stop with status 1 and name the field when either differs from `requires.secrets` or `requires.model_provider`. +Someone can edit the manifest in Git and leave `requires` as it was, so import and verify read the secret names and the model class, or null when the manifest has no chat model, out of the manifest again, and stop with status 1 and name the field when either differs from `requires.secrets` or `requires.model_provider`. Assistant verify checks the graph with `POST /assistants/search` filtered to the graph ID; every graph on a deployment has a default assistant, so an empty result means the deployment does not serve the graph. It checks each prompt in `requires.prompts`, such as `support-system:prod`, by asking LangSmith for the commit that tag points at (`GET /api/v1/commits/{owner}/{repo}/{tag}`, the request `langstar prompt pull --commit prod` already sends). A 404 means the workspace lacks the prompt or its tag. @@ -110,6 +110,7 @@ A mise task runs a short Python script that pulls the imported prompt with `incl - [langstar finds a deployment's Agent Server URL only in `source_config.custom_url` (`resolve_deployment_url` in `cli/src/deployment_utils.rs`), and a deployment built from GitHub has none, so every `langstar assistant` command, the new ones included, fails on such a deployment] → In [export-import-prompts-and-assistants Task 2.2 Return the URL of a deployment built from GitHub](tasks.md#L13), a developer adds the control plane's top-level `url` to the SDK's `Deployment`, which drops it today, and makes `resolve_deployment_url` use it, before anyone builds the assistant export, compare, verify and import commands. - [The SDK changes for export and import add public fields to `Deployment`, `CommitManifestResponse`, `CommitData`, `CommitRequest`, `Assistant` and the assistant request structs, so a crate that builds one of those structs with a struct literal stops compiling] → Each pull request that adds a field updates every struct literal of that struct in this repository. [Should the SDK field additions count as a breaking change?](#9---should-the-sdk-field-additions-count-as-a-breaking-change) asks codekiln whether those pull requests also carry a `BREAKING CHANGE:` footer, which `docs/dev/git-scm-conventions.md` says makes the release a major version. - [A tag in the destination may already point at another commit] → `--update` moves the tag, and `--dry-run` lists the move. +- [The control plane may refuse a second deployment from this repository with 409] → [export-import-prompts-and-assistants Task 2.3 Register the test graph under a second graph ID](tasks.md#L14) tries one before anyone builds the fixture, and a 409 sends [Which second workspace and second deployment should the tests use?](#5---which-second-workspace-and-second-deployment-should-the-tests-use) back to codekiln. ## Resolved Questions @@ -148,7 +149,7 @@ None. ### 5 - Which second workspace and second deployment should the tests use? > Context from the drafting agent, for question 5. -> The import tests need a workspace and a deployment to import into. The draft has codekiln create a second workspace in the test organization, which the existing API key can reach, and the tests read its ID from `LANGSTAR_TEST_DEST_WORKSPACE_ID`. The destination deployment fixture from [export-import-prompts-and-assistants Task 2.4 Add a destination deployment fixture](tasks.md#L15) deploys the test graph a second time from a `langgraph.json` of its own, which registers the graph as both `test_graph` and `test_graph_alt` so the update tests can move an assistant from one graph to the other. `.github/workflows/cleanup-test-deployments.yml` deletes each test deployment once it is four hours old and the fixture rebuilds it on the next CI run, so the first CI run after each cleanup waits for two deployments to build instead of one. Recommendation: a second workspace in the same organization, which codekiln creates, and one destination deployment that the fixture from task 2.4 rebuilds whenever the cleanup workflow has deleted it. +> The import tests need a workspace and a deployment to import into. The draft has codekiln create a second workspace in the test organization, which the existing API key can reach, and the tests read its ID from `LANGSTAR_TEST_DEST_WORKSPACE_ID`. The destination deployment fixture from [export-import-prompts-and-assistants Task 2.4 Add a destination deployment fixture](tasks.md#L15) deploys the test graph a second time from a `langgraph.json` of its own, which registers the graph as both `test_graph` and `test_graph_alt` so the update tests can move an assistant from one graph to the other. Nobody knows yet whether the control plane accepts that deployment. It answers a second deployment of the same agent environment with 409 "A deployment already exists for this agent environment", and no document says whether a second `langgraph.json` in the same repository makes a new agent environment. The check in [export-import-prompts-and-assistants Task 2.3 Register the test graph under a second graph ID](tasks.md#L14) is the first deployment that tries. If it gets the 409, the developer stops before task 2.4 and brings this question back to codekiln with what the control plane returned. `.github/workflows/cleanup-test-deployments.yml` deletes each test deployment once it is four hours old and the fixture rebuilds it on the next CI run, so the first CI run after each cleanup waits for two deployments to build instead of one. Recommendation: a second workspace in the same organization, which codekiln creates, and one destination deployment that the fixture from task 2.4 rebuilds whenever the cleanup workflow has deleted it. diff --git a/openspec/changes/export-import-prompts-and-assistants/proposal.md b/openspec/changes/export-import-prompts-and-assistants/proposal.md index e97833780..c1d08a591 100644 --- a/openspec/changes/export-import-prompts-and-assistants/proposal.md +++ b/openspec/changes/export-import-prompts-and-assistants/proposal.md @@ -32,7 +32,7 @@ A prompt belongs to a workspace and an assistant belongs to a deployment, but bo - SDK: `sdk/src/prompts.rs` gains a commit read that asks for the model and returns it on `CommitManifestResponse`, a commit description on commit create, the new commit's `id`, a repo update, a commit list that returns each commit's `id`, and calls that list, create and move commit tags. `sdk/src/assistants.rs` gains the `context` and `version` fields, `graph_id` on update, and a search by graph ID. `sdk/src/deployments.rs` gains the deployment's `url`. Each new public field breaks a crate outside this repository that builds that SDK struct with a struct literal; codekiln decides in [Should the SDK field additions count as a breaking change?](design.md#9---should-the-sdk-field-additions-count-as-a-breaking-change) whether that makes the release a major version. - CLI: new subcommands in `cli/src/commands/prompt.rs` and `cli/src/commands/assistant.rs`, and a shared module for the export file, the destination record, comparison and the exit status. `resolve_deployment_url` in `cli/src/deployment_utils.rs` also returns the Agent Server URL of a deployment built from GitHub, which every `langstar assistant` command needs. -- Tests need a second workspace, a second deployment and one model provider key for the structured output check. codekiln creates the workspace and the key, and langstar's tests read them through `fnox.toml` and the CI workflow; the destination deployment fixture from [export-import-prompts-and-assistants Task 2.4 Add a destination deployment fixture](tasks.md#L15) creates the deployment whenever the cleanup workflow has deleted it, as the source fixture does today. +- Tests need a second workspace, a second deployment and one model provider key for the structured output check. codekiln creates the workspace and the key, and langstar's tests read them through `fnox.toml` and the CI workflow; the destination deployment fixture from [export-import-prompts-and-assistants Task 2.4 Add a destination deployment fixture](tasks.md#L15) creates the deployment whenever the cleanup workflow has deleted it, as the source fixture does today. Nobody knows yet whether the control plane accepts a second deployment from this repository, and [export-import-prompts-and-assistants Task 2.3 Register the test graph under a second graph ID](tasks.md#L14) finds out before anyone builds the fixture. - Docs: a new usage page on moving prompts and assistants between environments. ## Citations diff --git a/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md index 86828f366..b9ff78825 100644 --- a/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md +++ b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md @@ -59,6 +59,17 @@ The export file SHALL list, under `requires.secrets`, the name of every secret t - **AND** a user runs `langstar prompt import my-prompt.json --workspace-id ` - **THEN** langstar creates no prompt, names `requires.secrets` and exits with status 1 +### Requirement: Export, import and verify a prompt without a model + +When the manifest holds no chat model constructor, `langstar prompt export` SHALL write `requires.model_provider` as null. Import and verify SHALL accept null when the manifest they read again holds no chat model, skip the saved model configuration check, report no note for it, and still run the secret check. + +#### Scenario: Prompt saved without a model + +- **WHEN** a user exports a prompt saved without a model and imports the file with `langstar prompt import my-prompt.json --workspace-id ` +- **THEN** the file holds null under `requires.model_provider` +- **AND** langstar creates the prompt without a saved model configuration check and exits with status 0 +- **AND** `langstar prompt verify my-prompt.json` against the destination reports no saved model configuration note and exits with status 0 + ### Requirement: Import requires --workspace-id `langstar prompt import ` SHALL require `--workspace-id`. It SHALL refuse to run when the flag is missing, even when `LANGSMITH_WORKSPACE_ID` is set, and exit with status 1 and an error naming the flag. @@ -202,7 +213,7 @@ After writing, `langstar prompt import` SHALL record the commit hash in the dest ### Requirement: Verify that a prompt can run in a workspace -`langstar prompt verify ` SHALL do what `compare` does and also check that the workspace has every secret in `requires.secrets`. It SHALL name each missing secret and exit with status 3 when any secret is missing. It SHALL also look for a saved model configuration whose settings `id` ends in the class name `requires.model_provider` holds, and report a missing one as a note that leaves the exit status unchanged, because the prompt runs with the model its manifest carries. +`langstar prompt verify ` SHALL do what `compare` does and also check that the workspace has every secret in `requires.secrets`. It SHALL name each missing secret and exit with status 3 when any secret is missing. When `requires.model_provider` is not null, it SHALL also look for a saved model configuration whose settings `id` ends in that class name, and report a missing one as a note that leaves the exit status unchanged, because the prompt runs with the model its manifest carries. #### Scenario: Secret missing in the destination diff --git a/openspec/changes/export-import-prompts-and-assistants/tasks.md b/openspec/changes/export-import-prompts-and-assistants/tasks.md index 56d5194ac..b5f2ac95a 100644 --- a/openspec/changes/export-import-prompts-and-assistants/tasks.md +++ b/openspec/changes/export-import-prompts-and-assistants/tasks.md @@ -11,7 +11,7 @@ Each numbered group below is one pull request, merged in the order listed, excep - [ ] 2.1 Add the destination workspace and the model provider key to `fnox.toml`, the CI workflow and `docs/dev/environment-variables.md`, with placeholders only; verify with `mise run check` and a CI run that reads both. This task waits on codekiln to create a second workspace in the test organization and an OpenAI API key, and to store both in 1Password and the repository's CI secrets and the key as the destination workspace's `OPENAI_API_KEY` secret. The import tests need the workspace as a place to import into, and its secret lets prompt import's secret check pass for a prompt with an OpenAI model. The key lets the check in 7.3 run each imported structured prompt once against a small OpenAI model, which bills that account on every CI run; it is the one test that shows a copy still returns structured data, where every other test compares text. - [ ] 2.2 Add `url` to `Deployment` in `sdk/src/deployments.rs`, with a method that returns `url` and falls back to `source_config.custom_url`, and make `resolve_deployment_url` in `cli/src/deployment_utils.rs` call that method, so that a deployment built from GitHub resolves; end the four `Deployment` struct literals in `sdk/tests/integration_deployment_workflow.rs` and `cli/src/commands/deployment.rs` with `..Default::default()`, because a crate that builds a `Deployment` with a struct literal stops compiling when the field is added; add a unit test for each case in `sdk/src/deployments.rs`, and verify by removing `#[ignore]` from `test_assistant_create_basic` in `cli/tests/assistant_command_test.rs` and running it against the test deployment -- [ ] 2.3 Add a second `langgraph.json` under `tests/fixtures/` that registers the test graph from `tests/fixtures/test-graph-deployment/langgraph.json` under two graph IDs, `test_graph` and `test_graph_alt`, so the update tests in 3.3 and 10.2 can move an assistant from one graph to the other. To verify, run `langstar deployment create` with `--branch` set to this pull request's branch and `--config-path` set to the new file while the source test deployment is live, run `langstar assistant list --deployment `, find one assistant with graph ID `test_graph` and one with `test_graph_alt`, then delete the deployment. +- [ ] 2.3 Add a second `langgraph.json` under `tests/fixtures/` that registers the test graph from `tests/fixtures/test-graph-deployment/langgraph.json` under two graph IDs, `test_graph` and `test_graph_alt`, so the update tests in 3.3 and 10.2 can move an assistant from one graph to the other. To verify, deploy the new file while the source test deployment is live, which shows whether the control plane accepts a second deployment from this repository: run `langstar deployment create` with `--branch` set to this pull request's branch and `--config-path` set to the new file. A 409 "A deployment already exists for this agent environment" means it does not, and the developer reports that to codekiln before anyone starts task 2.4. Otherwise run `langstar assistant list --deployment `, find one assistant with graph ID `test_graph` and one with `test_graph_alt`, then delete the deployment. - [ ] 2.4 Add a destination deployment fixture to `sdk/src/test_utils.rs` and `cli/tests/common/fixtures.rs` that deploys the `langgraph.json` from 2.3 under the name prefix `dest-integration-test`. `find_reusable_deployment` returns any deployment whose name starts with the fixture's prefix before it checks the graph config, so this prefix keeps the source fixture, whose prefix is `pr-integration-test`, from picking up the destination deployment. `find_reusable_deployment` then falls back to any live deployment built from the same repository and graph config file, so the separate file from 2.3 keeps the destination fixture from picking up the source deployment. The control plane answers a second deployment of one agent environment with 409 "A deployment already exists for this agent environment". No document says which fields make an agent environment, and the verification in 2.3 shows whether a second graph config file in the same repository makes a new one. `.github/workflows/cleanup-test-deployments.yml` runs every four hours and deletes each deployment whose name contains `integration-test` once it is four hours old or older, so the destination deployment is deleted and rebuilt on the same schedule as the source deployment. Verify with an integration test that asserts the destination and source deployment IDs differ and lists assistants on the destination deployment through the URL that 2.2 resolves ## 3. SDK reads and writes the fields export and import need @@ -32,25 +32,25 @@ Each task in this group adds a public field to one of these SDK structs: `Commit ## 5. prompt export -- [ ] 5.1 Add `langstar prompt export` with `--commit`, `--output` and the scoping flags; verify with integration tests that create prompts through the SDK (a `json_schema` prompt, a `function_calling` prompt with a nested schema, a prompt with several messages and a placeholder, and a prompt with a model and metadata), export each through the CLI, and assert every field of the file against the SDK's read of the commit, an export with `--commit prod` of a prompt whose `prod` tag points at an earlier commit, asserting that `source.commit_hash` and the manifest are that earlier commit's, and an error test that exporting a handle no prompt in the workspace has exits with status 1, names the handle and writes no file, and that the file for the prompt with a model lists `OPENAI_API_KEY` under `requires.secrets` and `ChatOpenAI` under `requires.model_provider` and holds no secret value +- [ ] 5.1 Add `langstar prompt export` with `--commit`, `--output` and the scoping flags; verify with integration tests that create prompts through the SDK (a `json_schema` prompt, a `function_calling` prompt with a nested schema, a prompt with several messages and a placeholder, and a prompt with a model and metadata), export each through the CLI, and assert every field of the file against the SDK's read of the commit, an export with `--commit prod` of a prompt whose `prod` tag points at an earlier commit, asserting that `source.commit_hash` and the manifest are that earlier commit's, and an error test that exporting a handle no prompt in the workspace has exits with status 1, names the handle and writes no file, and that the file for the prompt with a model lists `OPENAI_API_KEY` under `requires.secrets` and `ChatOpenAI` under `requires.model_provider` and holds no secret value, and that the file for the `json_schema` prompt, which has no model, holds an empty `requires.secrets` and null under `requires.model_provider` - [ ] 5.2 Document `prompt export` in `docs/usage/` and the README; verify that each documented command runs as written ## 6. prompt compare and verify - [ ] 6.1 Add `langstar prompt compare`; verify with integration tests that export a prompt, change its model temperature through the SDK, and assert the reported path, both values and exit status 3, and that a changed commit description and a changed repo readme are each reported the same way, and CLI tests that `langstar -f json prompt compare` prints one JSON object with `status`, `differences` and `checks` to stdout and its progress to stderr, and that `prompt compare` with a mistyped flag exits with status 2, and that `prompt compare` of a file holding `"format_version": 2` exits with status 1 and names `format_version` and the value 2 -- [ ] 6.2 Add `langstar prompt verify` with the secret check and the saved model configuration note, matching `requires.model_provider` against the class name at the end of each configuration's settings `id`; verify with integration tests against a workspace that lacks a listed secret and one that has it, a test that a missing saved model configuration is reported and leaves the exit status at 0, and a test that a file whose manifest refers to a secret its `requires.secrets` leaves out exits with status 1 and names `requires.secrets` +- [ ] 6.2 Add `langstar prompt verify` with the secret check and the saved model configuration note, matching `requires.model_provider` against the class name at the end of each configuration's settings `id`; verify with integration tests against a workspace that lacks a listed secret and one that has it, a test that a missing saved model configuration is reported and leaves the exit status at 0, a test that verify of a prompt without a model, whose `requires.model_provider` is null, reports no saved model configuration note and exits with status 0, and a test that a file whose manifest refers to a secret its `requires.secrets` leaves out exits with status 1 and names `requires.secrets` - [ ] 6.3 Document compare and verify; verify that each documented command runs as written ## 7. prompt import -- [ ] 7.1 Add `langstar prompt import`, which exits with status 1 and names `--workspace-id` when the flag is missing, runs the secret and saved model configuration checks of `prompt verify` from 6.2 before it writes, creates the repo, the commit and its tags, then reads the commit back; verify with integration tests that export from the test workspace, import into the destination workspace, and compare the full source and destination commits through the SDK with the model included, the commit description among them, and the repo's description, readme, tags and visibility, for every prompt shape from 5.1, that a file whose `requires.secrets` names a secret the destination workspace lacks creates no prompt, names the secret and exits with status 3, and a CLI test that runs import without `--workspace-id` while `LANGSMITH_WORKSPACE_ID` is set and asserts that it exits with status 1, names the missing flag and creates no prompt, and a CLI test that `prompt import` of an assistant export exits with status 1, names `kind` and the value `assistant`, and creates no prompt, and a CLI test that imports a file whose manifest refers to `ANTHROPIC_API_KEY` while `requires.secrets` holds only `OPENAI_API_KEY`, and another whose `requires.model_provider` names a class other than the one at the end of the manifest's chat model `id`, asserting that each creates no prompt, names the field and exits with status 1 +- [ ] 7.1 Add `langstar prompt import`, which exits with status 1 and names `--workspace-id` when the flag is missing, runs the secret and saved model configuration checks of `prompt verify` from 6.2 before it writes, creates the repo, the commit and its tags, then reads the commit back; verify with integration tests that export from the test workspace, import into the destination workspace, and compare the full source and destination commits through the SDK with the model included, the commit description among them, and the repo's description, readme, tags and visibility, for every prompt shape from 5.1, the prompt without a model among them, that a file whose `requires.secrets` names a secret the destination workspace lacks creates no prompt, names the secret and exits with status 3, and a CLI test that runs import without `--workspace-id` while `LANGSMITH_WORKSPACE_ID` is set and asserts that it exits with status 1, names the missing flag and creates no prompt, and a CLI test that `prompt import` of an assistant export exits with status 1, names `kind` and the value `assistant`, and creates no prompt, and a CLI test that imports a file whose manifest refers to `ANTHROPIC_API_KEY` while `requires.secrets` holds only `OPENAI_API_KEY`, and another whose `requires.model_provider` names a class other than the one at the end of the manifest's chat model `id`, asserting that each creates no prompt, names the field and exits with status 1 - [ ] 7.2 Add `--update`, `--dry-run`, `--record` and the destination record; verify with integration tests that a repeated import creates no commit, that a changed destination is refused without `--update`, that an import after someone pushed a commit to the destination since the recorded import says the destination changed since the last import and exits with status 3, that `--update` adds one commit with the right parent and the file's commit description, that `--update` against a destination whose repo description, readme, tags or visibility differ while the manifest matches updates the repo and adds no commit, that `--update` against a destination whose `prod` tag points at an earlier commit while the manifest matches moves the tag and adds no commit, that `--dry-run` lists the move, and that `--dry-run` writes nothing, and an httpmock test in which the commit read back lacks a field the file holds, asserting that import names that field's JSON Pointer path, exits with status 3, and leaves the new commit hash in the destination record - [ ] 7.3 Add the structured output check as a mise task that pulls the imported prompt with `include_model=True` and a `secrets` map holding the model provider key read from the environment, invokes it and validates the result against the schema; verify by running it from the import integration test for both methods - [ ] 7.4 Document prompt import, the workspace scope of prompts and setting up destination credentials; verify that each documented command runs as written ## 8. assistant export -- [ ] 8.1 Add `langstar assistant export` with `--output` and `--prompt-ref`; verify with integration tests that create an assistant with configuration, context and metadata through the SDK and assert every field of the exported file, an error test for a pointer that names nothing, and an error test that exporting an assistant ID the deployment lacks exits with status 1, names the ID and writes no file +- [ ] 8.1 Add `langstar assistant export` with `--output` and `--prompt-ref`; verify with integration tests that create an assistant with configuration, context and metadata through the SDK and assert every field of the exported file, that an export with `--prompt-ref /config/configurable/system_prompt` of an assistant whose value there is `support-system:prod` writes that pointer and `support-system:prod` under `requires.prompts`, an error test for a pointer that names nothing, and an error test that exporting an assistant ID the deployment lacks exits with status 1, names the ID and writes no file - [ ] 8.2 Document assistant export; verify that each documented command runs as written ## 9. assistant compare and verify From e04450e8524e1e5c8b4f0cfce930e4e91b8d5ac1 Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Tue, 13 Oct 2026 22:42:46 -0400 Subject: [PATCH 32/90] =?UTF-8?q?=F0=9F=93=9A=20docs(openspec):=20rename?= =?UTF-8?q?=20requires.model=5Fprovider=20to=20requires.model=5Fclass,=20u?= =?UTF-8?q?ntie=20two=20design=20sentences,=20and=20specify=20export=20to?= =?UTF-8?q?=20stdout,=20a=20missing=20copy=20at=20compare,=20and=20tag=20d?= =?UTF-8?q?elete?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Rename the export file's requires.model_provider to requires.model_class, so it no longer shares a name with LangSmith's commit.model_provider. Rewrite the design's two requires-check sentences so the action comes first. Add scenarios that compare exits with status 1 when the prompt or the recorded assistant is missing, require export without --output to print only the JSON document on stdout, and add a tag delete call to task 3.2 for the end-to-end test in task 10.3. #807 Export, import, compare, and verify prompts and assistants across LangSmith environments Co-Authored-By: Claude Opus 5.5 --- .../design.md | 12 ++++++---- .../proposal.md | 2 +- .../specs/assistant/spec.md | 14 ++++++++++- .../specs/prompt/spec.md | 24 ++++++++++++++----- .../tasks.md | 16 ++++++------- 5 files changed, 47 insertions(+), 21 deletions(-) diff --git a/openspec/changes/export-import-prompts-and-assistants/design.md b/openspec/changes/export-import-prompts-and-assistants/design.md index ab4f3d65b..817f35640 100644 --- a/openspec/changes/export-import-prompts-and-assistants/design.md +++ b/openspec/changes/export-import-prompts-and-assistants/design.md @@ -43,7 +43,7 @@ One prompt export looks like this: "source": { "workspace_id": "", "handle": "my-prompt", "commit_hash": "", "ref": "prod" }, "repo": { "handle": "my-prompt", "description": "...", "readme": "...", "tags": ["support"], "is_public": false }, "commit": { "description": "...", "manifest": { "lc": 1, "type": "constructor", "id": ["langsmith", "playground", "PromptPlayground"], "kwargs": { "first": {}, "last": {} } }, "model_config": {}, "model_provider": "...", "tags": ["prod"] }, - "requires": { "secrets": ["OPENAI_API_KEY"], "model_provider": "ChatOpenAI" } + "requires": { "secrets": ["OPENAI_API_KEY"], "model_class": "ChatOpenAI" } } ``` @@ -51,6 +51,8 @@ An assistant export has `kind: "assistant"`, a `source` with the deployment, ass The file holds no export time and langstar sorts its keys, so exporting an unchanged resource leaves Git with nothing to commit. +Without `--output`, export prints the file to stdout and nothing else, so `langstar prompt export my-prompt > my-prompt.json` writes a file import can read. Export writes its progress messages to stderr in every output format. [`OutputFormatter::print_message` in cli/src/output.rs:172](https://github.com/codekiln/langstar/blob/e4ae746d51f1f3f9ccf3fc97787148c3b4df2bce/cli/src/output.rs#L172) sends `info` to stdout in table mode, so export does not use it for those messages. + Import, compare and verify read a file only when its `format_version` is 1 and its `kind` matches the command: `prompt` for the prompt commands and `assistant` for the assistant commands. For any other file the command writes nothing, exits with status 1 and names the field and the value it found. When a user runs `prompt import` on an assistant export, or on a file a newer langstar wrote, langstar stops with that error before it changes anything in the destination. ### Export asks LangSmith for the model @@ -77,9 +79,9 @@ Compare reads the prompt or assistant from the destination, puts it in the shape Prompt verify checks that the workspace holds every secret the prompt's model reads. Export lists those names in `requires.secrets` by walking the manifest for LangChain secret references, `{"lc": 1, "type": "secret", "id": ["OPENAI_API_KEY"]}`, the shape the Python SDK's `_manifest_has_secrets` matches. Verify looks each name up among the workspace's secrets (`GET /api/v1/workspaces/current/secrets`, which `langstar secrets list` already calls) and prints the name of each missing secret and no secret value. -Prompt verify also notes when the workspace has no saved model configuration for the prompt's model, and leaves the exit status alone, because the prompt runs with the model its manifest carries. Export takes the model's class name, such as `ChatOpenAI`, from the last element of the chat model constructor's `id` in the manifest, such as `["langchain", "chat_models", "openai", "ChatOpenAI"]`, and stores it in `requires.model_provider`. Verify looks for a saved model configuration (`GET /api/v1/playground-settings`, which `langstar model-config list` calls) whose settings `id` ends in that class name. LangSmith also returns a `model_provider` field beside the manifest, which the file keeps as `commit.model_provider`; no SDK or spec shows what it holds, so in [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8) a developer records the value the live API returns. [#807 Export, import, compare, and verify prompts and assistants across LangSmith environments](https://github.com/codekiln/langstar/issues/807) asks for the note, and [Should prompt verify report a missing saved model configuration?](#7---should-prompt-verify-report-a-missing-saved-model-configuration) asks whether to keep it. A prompt saved without a model has no chat model constructor in its manifest, so export writes `requires.model_provider` as null, and verify and import skip the saved model configuration check for it and still run the secret check. +Prompt verify also notes when the workspace has no saved model configuration for the prompt's model, and leaves the exit status alone, because the prompt runs with the model its manifest carries. Export takes the model's class name, such as `ChatOpenAI`, from the last element of the chat model constructor's `id` in the manifest, such as `["langchain", "chat_models", "openai", "ChatOpenAI"]`, and stores it in `requires.model_class`. Verify looks for a saved model configuration (`GET /api/v1/playground-settings`, which `langstar model-config list` calls) whose settings `id` ends in that class name. LangSmith also returns a `model_provider` field beside the manifest, which the file keeps as `commit.model_provider`; no SDK or spec shows what it holds, so in [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8) a developer records the value the live API returns. [#807 Export, import, compare, and verify prompts and assistants across LangSmith environments](https://github.com/codekiln/langstar/issues/807) asks for the note, and [Should prompt verify report a missing saved model configuration?](#7---should-prompt-verify-report-a-missing-saved-model-configuration) asks whether to keep it. A prompt saved without a model has no chat model constructor in its manifest, so export writes `requires.model_class` as null, and verify and import skip the saved model configuration check for it and still run the secret check. -Someone can edit the manifest in Git and leave `requires` as it was, so import and verify read the secret names and the model class, or null when the manifest has no chat model, out of the manifest again, and stop with status 1 and name the field when either differs from `requires.secrets` or `requires.model_provider`. +Import and verify work out `requires` again from the file's manifest: the secret names, and the model class or null. When either differs from what the file holds, they write nothing, name the field and exit with status 1. This catches a manifest someone edited in Git without updating `requires`. Assistant verify checks the graph with `POST /assistants/search` filtered to the graph ID; every graph on a deployment has a default assistant, so an empty result means the deployment does not serve the graph. It checks each prompt in `requires.prompts`, such as `support-system:prod`, by asking LangSmith for the commit that tag points at (`GET /api/v1/commits/{owner}/{repo}/{tag}`, the request `langstar prompt pull --commit prod` already sends). A 404 means the workspace lacks the prompt or its tag. @@ -89,7 +91,7 @@ Before it writes anything, import checks that the destination has the secrets, t ### The user names an assistant's prompt references -An assistant refers to a prompt through whatever configuration key its graph reads, so langstar cannot find the references by itself. `assistant export --prompt-ref ` names each one. The assistant's configuration should name each prompt by handle and tag, such as `support-system:prod`, because the same prompt gets a different commit hash in each workspace. Import copies the commit tags to the destination workspace, so when the graph on the destination deployment asks LangSmith for `support-system:prod`, it gets the same prompt. Because `requires.prompts` repeats a value the configuration already holds, someone can edit one in Git and not the other, so compare, verify and import resolve each pointer against the file's `assistant` object when they read the file, and stop with status 1 when the value there differs from the recorded prompt name. +An assistant refers to a prompt through whatever configuration key its graph reads, so langstar cannot find the references by itself. `assistant export --prompt-ref ` names each one. The assistant's configuration should name each prompt by handle and tag, such as `support-system:prod`, because the same prompt gets a different commit hash in each workspace. Import copies the commit tags to the destination workspace, so when the graph on the destination deployment asks LangSmith for `support-system:prod`, it gets the same prompt. Compare, verify and import look up each pointer from `requires.prompts` in the file's `assistant` object, and exit with status 1 when the value there differs from the prompt name recorded beside the pointer. This catches a configuration someone edited in Git without updating `requires.prompts`. ### compare, verify and import exit with 3 when they find a difference @@ -163,7 +165,7 @@ None. ### 7 - Should prompt verify report a missing saved model configuration? > Context from the drafting agent, for question 7. -> [#807 Export, import, compare, and verify prompts and assistants across LangSmith environments](https://github.com/codekiln/langstar/issues/807) asks import to report missing model configurations, so the draft has prompt verify look for a saved model configuration (`GET /api/v1/playground-settings`) whose settings `id` ends in the class name the export file holds in `requires.model_provider`, such as `ChatOpenAI`. Nothing fails at run time without one, because the prompt's manifest carries the model and its settings; the secret check is the one that finds a prompt that cannot authenticate. Reporting it as a failure would make verify exit with 3 for a prompt that runs. Recommendation: report a missing saved model configuration as a note that leaves the exit status alone. +> [#807 Export, import, compare, and verify prompts and assistants across LangSmith environments](https://github.com/codekiln/langstar/issues/807) asks import to report missing model configurations, so the draft has prompt verify look for a saved model configuration (`GET /api/v1/playground-settings`) whose settings `id` ends in the class name the export file holds in `requires.model_class`, such as `ChatOpenAI`. Nothing fails at run time without one, because the prompt's manifest carries the model and its settings; the secret check is the one that finds a prompt that cannot authenticate. Reporting it as a failure would make verify exit with 3 for a prompt that runs. Recommendation: report a missing saved model configuration as a note that leaves the exit status alone. diff --git a/openspec/changes/export-import-prompts-and-assistants/proposal.md b/openspec/changes/export-import-prompts-and-assistants/proposal.md index c1d08a591..eba212470 100644 --- a/openspec/changes/export-import-prompts-and-assistants/proposal.md +++ b/openspec/changes/export-import-prompts-and-assistants/proposal.md @@ -30,7 +30,7 @@ A prompt belongs to a workspace and an assistant belongs to a deployment, but bo ## Impact -- SDK: `sdk/src/prompts.rs` gains a commit read that asks for the model and returns it on `CommitManifestResponse`, a commit description on commit create, the new commit's `id`, a repo update, a commit list that returns each commit's `id`, and calls that list, create and move commit tags. `sdk/src/assistants.rs` gains the `context` and `version` fields, `graph_id` on update, and a search by graph ID. `sdk/src/deployments.rs` gains the deployment's `url`. Each new public field breaks a crate outside this repository that builds that SDK struct with a struct literal; codekiln decides in [Should the SDK field additions count as a breaking change?](design.md#9---should-the-sdk-field-additions-count-as-a-breaking-change) whether that makes the release a major version. +- SDK: `sdk/src/prompts.rs` gains a commit read that asks for the model and returns it on `CommitManifestResponse`, a commit description on commit create, the new commit's `id`, a repo update, a commit list that returns each commit's `id`, calls that list, create and move commit tags, and a tag delete that the end-to-end test uses. `sdk/src/assistants.rs` gains the `context` and `version` fields, `graph_id` on update, and a search by graph ID. `sdk/src/deployments.rs` gains the deployment's `url`. Each new public field breaks a crate outside this repository that builds that SDK struct with a struct literal; codekiln decides in [Should the SDK field additions count as a breaking change?](design.md#9---should-the-sdk-field-additions-count-as-a-breaking-change) whether that makes the release a major version. - CLI: new subcommands in `cli/src/commands/prompt.rs` and `cli/src/commands/assistant.rs`, and a shared module for the export file, the destination record, comparison and the exit status. `resolve_deployment_url` in `cli/src/deployment_utils.rs` also returns the Agent Server URL of a deployment built from GitHub, which every `langstar assistant` command needs. - Tests need a second workspace, a second deployment and one model provider key for the structured output check. codekiln creates the workspace and the key, and langstar's tests read them through `fnox.toml` and the CI workflow; the destination deployment fixture from [export-import-prompts-and-assistants Task 2.4 Add a destination deployment fixture](tasks.md#L15) creates the deployment whenever the cleanup workflow has deleted it, as the source fixture does today. Nobody knows yet whether the control plane accepts a second deployment from this repository, and [export-import-prompts-and-assistants Task 2.3 Register the test graph under a second graph ID](tasks.md#L14) finds out before anyone builds the fixture. - Docs: a new usage page on moving prompts and assistants between environments. diff --git a/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md b/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md index d62b4df70..378bce84d 100644 --- a/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md +++ b/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md @@ -30,7 +30,7 @@ Every `langstar assistant` subcommand SHALL take a required `--deployment` flag ### Requirement: Export an assistant to a file -`langstar assistant export --deployment ` SHALL write one JSON file holding the assistant's name, description, graph ID, configuration, context and metadata, and a `source` object naming the deployment, assistant ID and version. `--output ` SHALL choose the file; without it the file goes to stdout. Exporting the same assistant version twice SHALL produce identical files. +`langstar assistant export --deployment ` SHALL write one JSON file holding the assistant's name, description, graph ID, configuration, context and metadata, and a `source` object naming the deployment, assistant ID and version. `--output ` SHALL choose the file; without it stdout SHALL hold only the file, and messages go to stderr. Exporting the same assistant version twice SHALL produce identical files. #### Scenario: Export an assistant @@ -43,6 +43,12 @@ Every `langstar assistant` subcommand SHALL take a required `--deployment` flag - **WHEN** the deployment has no assistant with that ID - **THEN** langstar exits with status 1 and an error naming the assistant ID +#### Scenario: Export to stdout + +- **WHEN** a user runs `langstar assistant export --deployment my-agent` without `--output` +- **THEN** stdout holds only the export file's JSON document +- **AND** any progress message goes to stderr + ### Requirement: Record the prompts an assistant refers to `langstar assistant export` SHALL accept `--prompt-ref `, repeatable, naming a string in the assistant's configuration or context that holds a prompt name. The file SHALL list each pointer and the prompt name it holds under `requires.prompts`. A pointer that resolves to no string SHALL be an error. @@ -161,6 +167,12 @@ After writing, `langstar assistant import` SHALL record the assistant ID in the - **WHEN** someone changed a metadata value on the deployment's assistant - **THEN** langstar lists that metadata path with both values and exits with status 3 +#### Scenario: No assistant to compare + +- **WHEN** the destination record has no entry for `my-agent-prod` +- **AND** a user runs `langstar assistant compare support-bot.json --deployment my-agent-prod` without `--assistant-id` +- **THEN** langstar exits with status 1 and an error that names the deployment and suggests `--assistant-id` + ### Requirement: Verify that an assistant can run on a deployment `langstar assistant verify --deployment ` SHALL do what `compare` does and also check that the deployment serves the file's graph ID and that the workspace, from `--workspace-id` or the configured workspace, has each prompt under `requires.prompts`. It SHALL name each missing graph and prompt and exit with status 3 when any check fails. diff --git a/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md index b9ff78825..71316235d 100644 --- a/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md +++ b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md @@ -6,7 +6,7 @@ ### Requirement: Export a prompt commit to a file -`langstar prompt export ` SHALL write one JSON file holding the prompt's repo details (handle, description, readme, tags, visibility), the commit's description, the commit's manifest as LangSmith returns it with the model included, the commit's model configuration and provider, the commit tags that point at it, and a `source` object naming the workspace, handle and commit hash. `--output ` SHALL choose the file; without it the file goes to stdout. +`langstar prompt export ` SHALL write one JSON file holding the repo details (handle, description, readme, tags, visibility), the commit's description, its manifest as LangSmith returns it with the model included, its model configuration and provider, the tags that point at it, and a `source` object naming the workspace, handle and commit hash. `--output ` SHALL choose the file; without it stdout SHALL hold only the file, and messages go to stderr. #### Scenario: Export the latest commit @@ -25,6 +25,12 @@ - **WHEN** no prompt in the workspace has the handle - **THEN** langstar exits with status 1 and an error naming the handle +#### Scenario: Export to stdout + +- **WHEN** a user runs `langstar prompt export my-prompt` without `--output` +- **THEN** stdout holds only the export file's JSON document +- **AND** any progress message goes to stderr + ### Requirement: Export a chosen commit or tag `langstar prompt export` SHALL accept `--commit ` and export that commit. Without the flag it SHALL export the latest commit. @@ -45,7 +51,7 @@ Exporting the same commit twice SHALL produce identical files. The file SHALL ho ### Requirement: List the credentials a prompt needs without their values -The export file SHALL list, under `requires.secrets`, the name of every secret the manifest refers to, and under `requires.model_provider`, the class name that ends the chat model constructor's `id`, such as `ChatOpenAI`. It SHALL hold no secret values. Import and verify SHALL read the secret names and the model class out of the file's manifest again, and when either differs from `requires.secrets` or `requires.model_provider`, write nothing, name the field and exit with status 1. +The export file SHALL list, under `requires.secrets`, the name of every secret the manifest refers to, and under `requires.model_class`, the class name that ends the chat model constructor's `id`, such as `ChatOpenAI`. It SHALL hold no secret values. Import and verify SHALL read the secret names and the model class out of the file's manifest again, and when either differs from `requires.secrets` or `requires.model_class`, write nothing, name the field and exit with status 1. #### Scenario: Model reads an API key secret @@ -61,12 +67,12 @@ The export file SHALL list, under `requires.secrets`, the name of every secret t ### Requirement: Export, import and verify a prompt without a model -When the manifest holds no chat model constructor, `langstar prompt export` SHALL write `requires.model_provider` as null. Import and verify SHALL accept null when the manifest they read again holds no chat model, skip the saved model configuration check, report no note for it, and still run the secret check. +When the manifest holds no chat model constructor, `langstar prompt export` SHALL write `requires.model_class` as null. Import and verify SHALL accept null when the manifest they read again holds no chat model, skip the saved model configuration check, report no note for it, and still run the secret check. #### Scenario: Prompt saved without a model - **WHEN** a user exports a prompt saved without a model and imports the file with `langstar prompt import my-prompt.json --workspace-id ` -- **THEN** the file holds null under `requires.model_provider` +- **THEN** the file holds null under `requires.model_class` - **AND** langstar creates the prompt without a saved model configuration check and exits with status 0 - **AND** `langstar prompt verify my-prompt.json` against the destination reports no saved model configuration note and exits with status 0 @@ -211,9 +217,15 @@ After writing, `langstar prompt import` SHALL record the commit hash in the dest - **WHEN** someone changed the readme of the workspace's prompt - **THEN** langstar lists the readme's JSON path with both values and exits with status 3 +#### Scenario: Prompt missing from the workspace + +- **WHEN** a user runs `langstar prompt compare my-prompt.json` +- **AND** the workspace has no prompt named `my-prompt` +- **THEN** langstar exits with status 1 and an error naming the handle + ### Requirement: Verify that a prompt can run in a workspace -`langstar prompt verify ` SHALL do what `compare` does and also check that the workspace has every secret in `requires.secrets`. It SHALL name each missing secret and exit with status 3 when any secret is missing. When `requires.model_provider` is not null, it SHALL also look for a saved model configuration whose settings `id` ends in that class name, and report a missing one as a note that leaves the exit status unchanged, because the prompt runs with the model its manifest carries. +`langstar prompt verify ` SHALL do what `compare` does and also check that the workspace has every secret in `requires.secrets`. It SHALL name each missing secret and exit with status 3 when any secret is missing. When `requires.model_class` is not null, it SHALL also look for a saved model configuration whose settings `id` ends in that class name, and report a missing one as a note that leaves the exit status unchanged, because the prompt runs with the model its manifest carries. #### Scenario: Secret missing in the destination @@ -223,7 +235,7 @@ After writing, `langstar prompt import` SHALL record the commit hash in the dest #### Scenario: No saved model configuration for the provider -- **WHEN** the file's `requires.model_provider` is `ChatOpenAI` +- **WHEN** the file's `requires.model_class` is `ChatOpenAI` - **AND** no saved model configuration in the destination workspace has a settings `id` ending in `ChatOpenAI` - **AND** every secret in `requires.secrets` exists and the prompt matches the file - **THEN** langstar notes that no saved model configuration matches `ChatOpenAI` and exits with status 0 diff --git a/openspec/changes/export-import-prompts-and-assistants/tasks.md b/openspec/changes/export-import-prompts-and-assistants/tasks.md index b5f2ac95a..5f519161a 100644 --- a/openspec/changes/export-import-prompts-and-assistants/tasks.md +++ b/openspec/changes/export-import-prompts-and-assistants/tasks.md @@ -19,7 +19,7 @@ Each numbered group below is one pull request, merged in the order listed, excep Each task in this group adds a public field to one of these SDK structs: `CommitManifestResponse` (the struct `PromptClient::get_commit` returns), `CommitData`, `CommitRequest`, `Assistant`, `CreateAssistantRequest`, `UpdateAssistantRequest` and `AssistantSearchRequest`. A crate that builds one of them with a struct literal stops compiling, so each task updates every struct literal of its struct in this repository. Whether this group's pull request and task 2.2's, which adds `url` to `Deployment`, carry a `BREAKING CHANGE:` footer waits on [Should the SDK field additions count as a breaking change?](design.md#9---should-the-sdk-field-additions-count-as-a-breaking-change). - [ ] 3.1 Add `description`, `model_config` and `model_provider` to `CommitManifestResponse`, and a commit read that sends `include=model` and returns that struct, to `sdk/src/prompts.rs`; verify with an httpmock test that asserts the query string and the parsed fields, and an integration test that reads a prompt with a model -- [ ] 3.2 Add the commit's `id` to `CommitData` in `sdk/src/prompts.rs`, a call that lists a prompt's commits with their `id` and hash (`GET /api/v1/commits/{owner}/{repo}`), and calls that list, create and move commit tags, which name their commit by that `id` in `POST /api/v1/repos/{owner}/{repo}/tags` and `PATCH /api/v1/repos/{owner}/{repo}/tags/{tag_name}`; verify with httpmock tests that assert `commit_id` in each request body, and an integration test that pushes two commits, tags the first, finds the second commit's `id` through the commit list, moves the tag to it, and reads the tag back through the SDK +- [ ] 3.2 Add the commit's `id` to `CommitData` in `sdk/src/prompts.rs`, a call that lists a prompt's commits with their `id` and hash (`GET /api/v1/commits/{owner}/{repo}`), calls that list, create and move commit tags, which name their commit by that `id` in `POST /api/v1/repos/{owner}/{repo}/tags` and `PATCH /api/v1/repos/{owner}/{repo}/tags/{tag_name}`, and a call that deletes a tag with `DELETE /api/v1/repos/{owner}/{repo}/tags/{tag_name}`, which task 10.3 uses; verify with httpmock tests that assert `commit_id` in each create and move request body and the path of the delete request, and an integration test that pushes two commits, tags the first, finds the second commit's `id` through the commit list, moves the tag to it, reads the tag back through the SDK, deletes it, and asserts that the tag list no longer holds it - [ ] 3.3 Add `context` and `version` to `Assistant`, `context` and `graph_id` to `UpdateAssistantRequest`, `context` to `CreateAssistantRequest`, and a `graph_id` filter to `AssistantSearchRequest`, in `sdk/src/assistants.rs`; verify with httpmock tests of the request bodies and an integration test that creates an assistant with context on the destination deployment from 2.4, reads it back with its version, finds it by graph ID, moves it from `test_graph` to `test_graph_alt`, and reads `test_graph_alt` back. The test runs on the destination deployment because the source deployment serves only `test_graph`, and task 2.4, which adds the destination fixture, merges before this group. - [ ] 3.4 Add `description` to `CommitRequest`, and a repo update that sends `PATCH /api/v1/repos/{owner}/{repo}` with `description`, `readme`, `tags` and `is_public`, to `sdk/src/prompts.rs`; verify with httpmock tests of the request bodies and an integration test that pushes a commit with a description, updates the repo's description, readme, tags and visibility, and reads each back @@ -32,30 +32,30 @@ Each task in this group adds a public field to one of these SDK structs: `Commit ## 5. prompt export -- [ ] 5.1 Add `langstar prompt export` with `--commit`, `--output` and the scoping flags; verify with integration tests that create prompts through the SDK (a `json_schema` prompt, a `function_calling` prompt with a nested schema, a prompt with several messages and a placeholder, and a prompt with a model and metadata), export each through the CLI, and assert every field of the file against the SDK's read of the commit, an export with `--commit prod` of a prompt whose `prod` tag points at an earlier commit, asserting that `source.commit_hash` and the manifest are that earlier commit's, and an error test that exporting a handle no prompt in the workspace has exits with status 1, names the handle and writes no file, and that the file for the prompt with a model lists `OPENAI_API_KEY` under `requires.secrets` and `ChatOpenAI` under `requires.model_provider` and holds no secret value, and that the file for the `json_schema` prompt, which has no model, holds an empty `requires.secrets` and null under `requires.model_provider` +- [ ] 5.1 Add `langstar prompt export` with `--commit`, `--output` and the scoping flags; verify with integration tests that create prompts through the SDK (a `json_schema` prompt, a `function_calling` prompt with a nested schema, a prompt with several messages and a placeholder, and a prompt with a model and metadata), export each through the CLI, and assert every field of the file against the SDK's read of the commit, an export with `--commit prod` of a prompt whose `prod` tag points at an earlier commit, asserting that `source.commit_hash` and the manifest are that earlier commit's, and an error test that exporting a handle no prompt in the workspace has exits with status 1, names the handle and writes no file, and that the file for the prompt with a model lists `OPENAI_API_KEY` under `requires.secrets` and `ChatOpenAI` under `requires.model_class` and holds no secret value, and that the file for the `json_schema` prompt, which has no model, holds an empty `requires.secrets` and null under `requires.model_class`, and a CLI test that runs `prompt export` without `--output`, parses all of stdout as one JSON document, and asserts that its `source.commit_hash` matches the SDK's read and that any progress message is on stderr - [ ] 5.2 Document `prompt export` in `docs/usage/` and the README; verify that each documented command runs as written ## 6. prompt compare and verify -- [ ] 6.1 Add `langstar prompt compare`; verify with integration tests that export a prompt, change its model temperature through the SDK, and assert the reported path, both values and exit status 3, and that a changed commit description and a changed repo readme are each reported the same way, and CLI tests that `langstar -f json prompt compare` prints one JSON object with `status`, `differences` and `checks` to stdout and its progress to stderr, and that `prompt compare` with a mistyped flag exits with status 2, and that `prompt compare` of a file holding `"format_version": 2` exits with status 1 and names `format_version` and the value 2 -- [ ] 6.2 Add `langstar prompt verify` with the secret check and the saved model configuration note, matching `requires.model_provider` against the class name at the end of each configuration's settings `id`; verify with integration tests against a workspace that lacks a listed secret and one that has it, a test that a missing saved model configuration is reported and leaves the exit status at 0, a test that verify of a prompt without a model, whose `requires.model_provider` is null, reports no saved model configuration note and exits with status 0, and a test that a file whose manifest refers to a secret its `requires.secrets` leaves out exits with status 1 and names `requires.secrets` +- [ ] 6.1 Add `langstar prompt compare`; verify with integration tests that export a prompt, change its model temperature through the SDK, and assert the reported path, both values and exit status 3, and that a changed commit description and a changed repo readme are each reported the same way, and CLI tests that `langstar -f json prompt compare` prints one JSON object with `status`, `differences` and `checks` to stdout and its progress to stderr, and that `prompt compare` with a mistyped flag exits with status 2, and that `prompt compare` of a file holding `"format_version": 2` exits with status 1 and names `format_version` and the value 2, and that `prompt compare` of a file whose handle the workspace lacks exits with status 1 and names the handle +- [ ] 6.2 Add `langstar prompt verify` with the secret check and the saved model configuration note, matching `requires.model_class` against the class name at the end of each configuration's settings `id`; verify with integration tests against a workspace that lacks a listed secret and one that has it, a test that a missing saved model configuration is reported and leaves the exit status at 0, a test that verify of a prompt without a model, whose `requires.model_class` is null, reports no saved model configuration note and exits with status 0, and a test that a file whose manifest refers to a secret its `requires.secrets` leaves out exits with status 1 and names `requires.secrets` - [ ] 6.3 Document compare and verify; verify that each documented command runs as written ## 7. prompt import -- [ ] 7.1 Add `langstar prompt import`, which exits with status 1 and names `--workspace-id` when the flag is missing, runs the secret and saved model configuration checks of `prompt verify` from 6.2 before it writes, creates the repo, the commit and its tags, then reads the commit back; verify with integration tests that export from the test workspace, import into the destination workspace, and compare the full source and destination commits through the SDK with the model included, the commit description among them, and the repo's description, readme, tags and visibility, for every prompt shape from 5.1, the prompt without a model among them, that a file whose `requires.secrets` names a secret the destination workspace lacks creates no prompt, names the secret and exits with status 3, and a CLI test that runs import without `--workspace-id` while `LANGSMITH_WORKSPACE_ID` is set and asserts that it exits with status 1, names the missing flag and creates no prompt, and a CLI test that `prompt import` of an assistant export exits with status 1, names `kind` and the value `assistant`, and creates no prompt, and a CLI test that imports a file whose manifest refers to `ANTHROPIC_API_KEY` while `requires.secrets` holds only `OPENAI_API_KEY`, and another whose `requires.model_provider` names a class other than the one at the end of the manifest's chat model `id`, asserting that each creates no prompt, names the field and exits with status 1 +- [ ] 7.1 Add `langstar prompt import`, which exits with status 1 and names `--workspace-id` when the flag is missing, runs the secret and saved model configuration checks of `prompt verify` from 6.2 before it writes, creates the repo, the commit and its tags, then reads the commit back; verify with integration tests that export from the test workspace, import into the destination workspace, and compare the full source and destination commits through the SDK with the model included, the commit description among them, and the repo's description, readme, tags and visibility, for every prompt shape from 5.1, the prompt without a model among them, that a file whose `requires.secrets` names a secret the destination workspace lacks creates no prompt, names the secret and exits with status 3, and a CLI test that runs import without `--workspace-id` while `LANGSMITH_WORKSPACE_ID` is set and asserts that it exits with status 1, names the missing flag and creates no prompt, and a CLI test that `prompt import` of an assistant export exits with status 1, names `kind` and the value `assistant`, and creates no prompt, and a CLI test that imports a file whose manifest refers to `ANTHROPIC_API_KEY` while `requires.secrets` holds only `OPENAI_API_KEY`, and another whose `requires.model_class` names a class other than the one at the end of the manifest's chat model `id`, asserting that each creates no prompt, names the field and exits with status 1 - [ ] 7.2 Add `--update`, `--dry-run`, `--record` and the destination record; verify with integration tests that a repeated import creates no commit, that a changed destination is refused without `--update`, that an import after someone pushed a commit to the destination since the recorded import says the destination changed since the last import and exits with status 3, that `--update` adds one commit with the right parent and the file's commit description, that `--update` against a destination whose repo description, readme, tags or visibility differ while the manifest matches updates the repo and adds no commit, that `--update` against a destination whose `prod` tag points at an earlier commit while the manifest matches moves the tag and adds no commit, that `--dry-run` lists the move, and that `--dry-run` writes nothing, and an httpmock test in which the commit read back lacks a field the file holds, asserting that import names that field's JSON Pointer path, exits with status 3, and leaves the new commit hash in the destination record - [ ] 7.3 Add the structured output check as a mise task that pulls the imported prompt with `include_model=True` and a `secrets` map holding the model provider key read from the environment, invokes it and validates the result against the schema; verify by running it from the import integration test for both methods - [ ] 7.4 Document prompt import, the workspace scope of prompts and setting up destination credentials; verify that each documented command runs as written ## 8. assistant export -- [ ] 8.1 Add `langstar assistant export` with `--output` and `--prompt-ref`; verify with integration tests that create an assistant with configuration, context and metadata through the SDK and assert every field of the exported file, that an export with `--prompt-ref /config/configurable/system_prompt` of an assistant whose value there is `support-system:prod` writes that pointer and `support-system:prod` under `requires.prompts`, an error test for a pointer that names nothing, and an error test that exporting an assistant ID the deployment lacks exits with status 1, names the ID and writes no file +- [ ] 8.1 Add `langstar assistant export` with `--output` and `--prompt-ref`; verify with integration tests that create an assistant with configuration, context and metadata through the SDK and assert every field of the exported file, that an export with `--prompt-ref /config/configurable/system_prompt` of an assistant whose value there is `support-system:prod` writes that pointer and `support-system:prod` under `requires.prompts`, an error test for a pointer that names nothing, an error test that exporting an assistant ID the deployment lacks exits with status 1, names the ID and writes no file, and a CLI test that runs `assistant export` without `--output`, parses all of stdout as one JSON document, and asserts that its `source.assistant_id` is the exported assistant's ID and that any progress message is on stderr - [ ] 8.2 Document assistant export; verify that each documented command runs as written ## 9. assistant compare and verify -- [ ] 9.1 Add `langstar assistant compare` with `--assistant-id`, and the lookup of the recorded assistant in the destination record from 4.1; verify with an integration test that exports an assistant, changes its metadata through the SDK, runs compare with `--assistant-id` set to that assistant, and asserts the reported path and exit status 3, and a unit test that compare reads the assistant ID from a destination record written by the 4.1 types; add the check, shared by assistant compare, verify and import, that each pointer under `requires.prompts` still resolves in the file's `assistant` object to the prompt name recorded with it, and verify with a unit test that a file whose configuration names another prompt at that pointer exits with status 1 and names the pointer and both values +- [ ] 9.1 Add `langstar assistant compare` with `--assistant-id`, and the lookup of the recorded assistant in the destination record from 4.1; verify with an integration test that exports an assistant, changes its metadata through the SDK, runs compare with `--assistant-id` set to that assistant, and asserts the reported path and exit status 3, and a unit test that compare reads the assistant ID from a destination record written by the 4.1 types, and an integration test that `assistant compare` with no destination record entry and no `--assistant-id` exits with status 1 and names the deployment; add the check, shared by assistant compare, verify and import, that each pointer under `requires.prompts` still resolves in the file's `assistant` object to the prompt name recorded with it, and verify with a unit test that a file whose configuration names another prompt at that pointer exits with status 1 and names the pointer and both values - [ ] 9.2 Add `langstar assistant verify` with `--assistant-id`, `--workspace-id` for the workspace that holds the prompts, the graph check, and the prompt check, which reads `GET /api/v1/commits/{owner}/{repo}/{tag}`; verify with integration tests that run verify with `--assistant-id` against an exported assistant, one with a file whose graph ID the deployment lacks, one with a missing prompt tag, and one that passes `--workspace-id` set to the destination workspace, and a CLI test that `langstar -f json assistant verify` prints one JSON object with `status`, `differences` and `checks` to stdout - [ ] 9.3 Document assistant compare and verify; verify that each documented command runs as written @@ -63,7 +63,7 @@ Each task in this group adds a public field to one of these SDK structs: `Commit - [ ] 10.1 Add `langstar assistant import` with `--workspace-id`, the graph check and prompt check of `assistant verify` from 9.2 run before it writes, creation, read-back, the destination record and `--record`; verify with an integration test that creates a tagged prompt in the test workspace, imports an assistant that refers to it into the destination deployment and checks name, graph ID, configuration, context, metadata and prompt references through the Agent Server API, that `--record ` writes the destination record to that path, and that a file whose graph ID the deployment lacks and a file with a missing prompt tag each create no assistant and exit with status 3, and an httpmock test in which the assistant read back differs from the file, asserting that import exits with status 3, that the destination record holds the new assistant ID, and that a second import creates no assistant, and a CLI test that `assistant import` of a prompt export exits with status 1, names `kind` and the value `prompt`, and creates no assistant - [ ] 10.2 Add `--update`, `--assistant-id` and `--dry-run`; verify with integration tests that a repeated import creates no assistant, that `--update` changes the recorded one, moving it from `test_graph` to `test_graph_alt` among the fields, that `--update --assistant-id ` naming an assistant other than the recorded one updates that assistant, leaves the recorded one unchanged and writes the new ID to the destination record, that `--update` of an assistant with metadata `{"kept": 1, "stale": 2}` from a file with `{"kept": 1}` leaves `{"kept": 1}` or reports `/metadata/stale` and exits with status 3, whichever task 1.2 found the server does, that `--update` of an assistant with a description and context from a file with neither clears both or reports `/description` and `/context` and exits with status 3, whichever task 1.2 found the server does, that a deleted recorded assistant exits with status 1, and that `--dry-run` creates no assistant, changes none and leaves the destination record as it was -- [ ] 10.3 Add an end-to-end test that imports prompts into the destination workspace, then imports an assistant that refers to them onto the destination deployment and runs `assistant verify`, passing `--workspace-id` set to the destination workspace to both assistant commands; verify that it passes, and that verify fails after deleting one prompt tag in the destination workspace +- [ ] 10.3 Add an end-to-end test that imports prompts into the destination workspace, then imports an assistant that refers to them onto the destination deployment and runs `assistant verify`, passing `--workspace-id` set to the destination workspace to both assistant commands; verify that it passes, and that verify fails after deleting one prompt tag in the destination workspace through the tag delete call from 3.2 - [ ] 10.4 Document assistant import, the deployment scope of assistants, and how export and import relate to the `create`, `get` and `update` commands of [#728 Implement CRUD command structure (create, get, update) for prompt commands](https://github.com/codekiln/langstar/issues/728); verify that each documented command runs as written ## Workflow follow-up From 1db48077436fea3a0e953378300007e3848a6f65 Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Tue, 13 Oct 2026 22:48:39 -0400 Subject: [PATCH 33/90] =?UTF-8?q?=F0=9F=93=9A=20docs(openspec):=20make=20D?= =?UTF-8?q?eployment.url=20optional,=20and=20add=20the=20repo's=20readme?= =?UTF-8?q?=20and=20tags=20to=20Prompt?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The control-plane spec declares Deployment.url nullable, so task 2.2 adds it as Option with serde default and tests a null and a missing url. PromptClient::get returns Prompt, which has no readme or repo tags, so task 3.4 adds them and reads them back through get, and question 9 and the proposal list Prompt among the structs that gain fields. #807 Export, import, compare, and verify prompts and assistants across LangSmith environments Co-Authored-By: Claude Opus 5.5 --- .../changes/export-import-prompts-and-assistants/design.md | 2 +- .../changes/export-import-prompts-and-assistants/proposal.md | 2 +- .../specs/assistant/spec.md | 2 +- .../changes/export-import-prompts-and-assistants/tasks.md | 4 ++-- 4 files changed, 5 insertions(+), 5 deletions(-) diff --git a/openspec/changes/export-import-prompts-and-assistants/design.md b/openspec/changes/export-import-prompts-and-assistants/design.md index 817f35640..f5921e6ef 100644 --- a/openspec/changes/export-import-prompts-and-assistants/design.md +++ b/openspec/changes/export-import-prompts-and-assistants/design.md @@ -179,6 +179,6 @@ None. ### 9 - Should the SDK field additions count as a breaking change? > Context from the drafting agent, for question 9. -> A new public field breaks only a crate that builds the struct with a struct literal. The only crate known to use the SDK is langstar's own CLI, and each pull request that adds a field updates the CLI's struct literals. `sdk/Cargo.toml` leaves the crate publishable, though, so an outside crate could break, and `docs/dev/git-scm-conventions.md` gives every API break a `BREAKING CHANGE:` footer, which makes the release a major version. The structs that gain fields are `Deployment`, `CommitManifestResponse` (which `PromptClient::get_commit` returns), `CommitData`, `CommitRequest`, `Assistant` and the assistant request structs. Marking them `#[non_exhaustive]` would stop every crate except the SDK itself from building them with struct literals, even with `..Default::default()`. langstar's own CLI builds `Assistant`, `CommitRequest`, `CreateAssistantRequest`, `UpdateAssistantRequest` and `Deployment` with struct literals today, and the SDK's integration tests build all of them except `Assistant` the same way, so the SDK would need a constructor for each of those five structs before the CLI and those tests compile again. After that one break, a new field would break nothing. [🚨 BREAKING CHANGE(sdk): move annotation-queue run calls to the platform items API · Pull Request #815](https://github.com/codekiln/langstar/pull/815) removes SDK functions and changes return types, marks itself breaking, and leaves that choice to codekiln too, so one answer to this question can set the policy for both pull requests. Recommendation: treat the additions as `feat` changes with no `BREAKING CHANGE:` footer, and name the new fields in the release notes. +> A new public field breaks only a crate that builds the struct with a struct literal. The only crate known to use the SDK is langstar's own CLI, and each pull request that adds a field updates the CLI's struct literals. `sdk/Cargo.toml` leaves the crate publishable, though, so an outside crate could break, and `docs/dev/git-scm-conventions.md` gives every API break a `BREAKING CHANGE:` footer, which makes the release a major version. The structs that gain fields are `Deployment`, `Prompt` (which `PromptClient::get` returns), `CommitManifestResponse` (which `PromptClient::get_commit` returns), `CommitData`, `CommitRequest`, `Assistant` and the assistant request structs. Marking them `#[non_exhaustive]` would stop every crate except the SDK itself from building them with struct literals, even with `..Default::default()`. langstar's own CLI builds `Assistant`, `CommitRequest`, `CreateAssistantRequest`, `UpdateAssistantRequest` and `Deployment` with struct literals today, and the SDK's integration tests build all of them except `Assistant` the same way, so the SDK would need a constructor for each of those five structs before the CLI and those tests compile again. After that one break, a new field would break nothing. [🚨 BREAKING CHANGE(sdk): move annotation-queue run calls to the platform items API · Pull Request #815](https://github.com/codekiln/langstar/pull/815) removes SDK functions and changes return types, marks itself breaking, and leaves that choice to codekiln too, so one answer to this question can set the policy for both pull requests. Recommendation: treat the additions as `feat` changes with no `BREAKING CHANGE:` footer, and name the new fields in the release notes. diff --git a/openspec/changes/export-import-prompts-and-assistants/proposal.md b/openspec/changes/export-import-prompts-and-assistants/proposal.md index eba212470..d165ab2cf 100644 --- a/openspec/changes/export-import-prompts-and-assistants/proposal.md +++ b/openspec/changes/export-import-prompts-and-assistants/proposal.md @@ -30,7 +30,7 @@ A prompt belongs to a workspace and an assistant belongs to a deployment, but bo ## Impact -- SDK: `sdk/src/prompts.rs` gains a commit read that asks for the model and returns it on `CommitManifestResponse`, a commit description on commit create, the new commit's `id`, a repo update, a commit list that returns each commit's `id`, calls that list, create and move commit tags, and a tag delete that the end-to-end test uses. `sdk/src/assistants.rs` gains the `context` and `version` fields, `graph_id` on update, and a search by graph ID. `sdk/src/deployments.rs` gains the deployment's `url`. Each new public field breaks a crate outside this repository that builds that SDK struct with a struct literal; codekiln decides in [Should the SDK field additions count as a breaking change?](design.md#9---should-the-sdk-field-additions-count-as-a-breaking-change) whether that makes the release a major version. +- SDK: `sdk/src/prompts.rs` gains a commit read that asks for the model and returns it on `CommitManifestResponse`, a commit description on commit create, the new commit's `id`, the repo's readme and tags on `Prompt`, a repo update, a commit list that returns each commit's `id`, calls that list, create and move commit tags, and a tag delete that the end-to-end test uses. `sdk/src/assistants.rs` gains the `context` and `version` fields, `graph_id` on update, and a search by graph ID. `sdk/src/deployments.rs` gains the deployment's `url`. Each new public field breaks a crate outside this repository that builds that SDK struct with a struct literal; codekiln decides in [Should the SDK field additions count as a breaking change?](design.md#9---should-the-sdk-field-additions-count-as-a-breaking-change) whether that makes the release a major version. - CLI: new subcommands in `cli/src/commands/prompt.rs` and `cli/src/commands/assistant.rs`, and a shared module for the export file, the destination record, comparison and the exit status. `resolve_deployment_url` in `cli/src/deployment_utils.rs` also returns the Agent Server URL of a deployment built from GitHub, which every `langstar assistant` command needs. - Tests need a second workspace, a second deployment and one model provider key for the structured output check. codekiln creates the workspace and the key, and langstar's tests read them through `fnox.toml` and the CI workflow; the destination deployment fixture from [export-import-prompts-and-assistants Task 2.4 Add a destination deployment fixture](tasks.md#L15) creates the deployment whenever the cleanup workflow has deleted it, as the source fixture does today. Nobody knows yet whether the control plane accepts a second deployment from this repository, and [export-import-prompts-and-assistants Task 2.3 Register the test graph under a second graph ID](tasks.md#L14) finds out before anyone builds the fixture. - Docs: a new usage page on moving prompts and assistants between environments. diff --git a/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md b/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md index 378bce84d..d0ef37e6c 100644 --- a/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md +++ b/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md @@ -2,7 +2,7 @@ ### Requirement: Every assistant command targets one deployment -Every `langstar assistant` subcommand SHALL take a required `--deployment` flag holding a deployment name or ID. Langstar SHALL look the deployment up through the control plane API and send the assistant request to the deployment's `url`, or to its `source_config.custom_url` when `url` is empty. +Every `langstar assistant` subcommand SHALL take a required `--deployment` flag holding a deployment name or ID. Langstar SHALL look the deployment up through the control plane API and send the assistant request to the deployment's `url`, or to its `source_config.custom_url` when `url` is null, missing or empty. #### Scenario: Deployment found by name diff --git a/openspec/changes/export-import-prompts-and-assistants/tasks.md b/openspec/changes/export-import-prompts-and-assistants/tasks.md index 5f519161a..d51b0237f 100644 --- a/openspec/changes/export-import-prompts-and-assistants/tasks.md +++ b/openspec/changes/export-import-prompts-and-assistants/tasks.md @@ -10,7 +10,7 @@ Each numbered group below is one pull request, merged in the order listed, excep ## 2. Test environments - [ ] 2.1 Add the destination workspace and the model provider key to `fnox.toml`, the CI workflow and `docs/dev/environment-variables.md`, with placeholders only; verify with `mise run check` and a CI run that reads both. This task waits on codekiln to create a second workspace in the test organization and an OpenAI API key, and to store both in 1Password and the repository's CI secrets and the key as the destination workspace's `OPENAI_API_KEY` secret. The import tests need the workspace as a place to import into, and its secret lets prompt import's secret check pass for a prompt with an OpenAI model. The key lets the check in 7.3 run each imported structured prompt once against a small OpenAI model, which bills that account on every CI run; it is the one test that shows a copy still returns structured data, where every other test compares text. -- [ ] 2.2 Add `url` to `Deployment` in `sdk/src/deployments.rs`, with a method that returns `url` and falls back to `source_config.custom_url`, and make `resolve_deployment_url` in `cli/src/deployment_utils.rs` call that method, so that a deployment built from GitHub resolves; end the four `Deployment` struct literals in `sdk/tests/integration_deployment_workflow.rs` and `cli/src/commands/deployment.rs` with `..Default::default()`, because a crate that builds a `Deployment` with a struct literal stops compiling when the field is added; add a unit test for each case in `sdk/src/deployments.rs`, and verify by removing `#[ignore]` from `test_assistant_create_basic` in `cli/tests/assistant_command_test.rs` and running it against the test deployment +- [ ] 2.2 Add `url` to `Deployment` in `sdk/src/deployments.rs` as an `Option` with `#[serde(default)]`, because the control-plane OpenAPI spec declares it nullable and a plain `String` would make a deployment list holding an unprovisioned deployment fail to parse, with a method that returns `url` and falls back to `source_config.custom_url` when `url` is null, missing or empty, and make `resolve_deployment_url` in `cli/src/deployment_utils.rs` call that method, so that a deployment built from GitHub resolves; end the four `Deployment` struct literals in `sdk/tests/integration_deployment_workflow.rs` and `cli/src/commands/deployment.rs` with `..Default::default()`, because a crate that builds a `Deployment` with a struct literal stops compiling when the field is added; add a unit test for each case in `sdk/src/deployments.rs`, among them a deployment whose `url` is null and one whose JSON has no `url` key, and verify by removing `#[ignore]` from `test_assistant_create_basic` in `cli/tests/assistant_command_test.rs` and running it against the test deployment - [ ] 2.3 Add a second `langgraph.json` under `tests/fixtures/` that registers the test graph from `tests/fixtures/test-graph-deployment/langgraph.json` under two graph IDs, `test_graph` and `test_graph_alt`, so the update tests in 3.3 and 10.2 can move an assistant from one graph to the other. To verify, deploy the new file while the source test deployment is live, which shows whether the control plane accepts a second deployment from this repository: run `langstar deployment create` with `--branch` set to this pull request's branch and `--config-path` set to the new file. A 409 "A deployment already exists for this agent environment" means it does not, and the developer reports that to codekiln before anyone starts task 2.4. Otherwise run `langstar assistant list --deployment `, find one assistant with graph ID `test_graph` and one with `test_graph_alt`, then delete the deployment. - [ ] 2.4 Add a destination deployment fixture to `sdk/src/test_utils.rs` and `cli/tests/common/fixtures.rs` that deploys the `langgraph.json` from 2.3 under the name prefix `dest-integration-test`. `find_reusable_deployment` returns any deployment whose name starts with the fixture's prefix before it checks the graph config, so this prefix keeps the source fixture, whose prefix is `pr-integration-test`, from picking up the destination deployment. `find_reusable_deployment` then falls back to any live deployment built from the same repository and graph config file, so the separate file from 2.3 keeps the destination fixture from picking up the source deployment. The control plane answers a second deployment of one agent environment with 409 "A deployment already exists for this agent environment". No document says which fields make an agent environment, and the verification in 2.3 shows whether a second graph config file in the same repository makes a new one. `.github/workflows/cleanup-test-deployments.yml` runs every four hours and deletes each deployment whose name contains `integration-test` once it is four hours old or older, so the destination deployment is deleted and rebuilt on the same schedule as the source deployment. Verify with an integration test that asserts the destination and source deployment IDs differ and lists assistants on the destination deployment through the URL that 2.2 resolves @@ -21,7 +21,7 @@ Each task in this group adds a public field to one of these SDK structs: `Commit - [ ] 3.1 Add `description`, `model_config` and `model_provider` to `CommitManifestResponse`, and a commit read that sends `include=model` and returns that struct, to `sdk/src/prompts.rs`; verify with an httpmock test that asserts the query string and the parsed fields, and an integration test that reads a prompt with a model - [ ] 3.2 Add the commit's `id` to `CommitData` in `sdk/src/prompts.rs`, a call that lists a prompt's commits with their `id` and hash (`GET /api/v1/commits/{owner}/{repo}`), calls that list, create and move commit tags, which name their commit by that `id` in `POST /api/v1/repos/{owner}/{repo}/tags` and `PATCH /api/v1/repos/{owner}/{repo}/tags/{tag_name}`, and a call that deletes a tag with `DELETE /api/v1/repos/{owner}/{repo}/tags/{tag_name}`, which task 10.3 uses; verify with httpmock tests that assert `commit_id` in each create and move request body and the path of the delete request, and an integration test that pushes two commits, tags the first, finds the second commit's `id` through the commit list, moves the tag to it, reads the tag back through the SDK, deletes it, and asserts that the tag list no longer holds it - [ ] 3.3 Add `context` and `version` to `Assistant`, `context` and `graph_id` to `UpdateAssistantRequest`, `context` to `CreateAssistantRequest`, and a `graph_id` filter to `AssistantSearchRequest`, in `sdk/src/assistants.rs`; verify with httpmock tests of the request bodies and an integration test that creates an assistant with context on the destination deployment from 2.4, reads it back with its version, finds it by graph ID, moves it from `test_graph` to `test_graph_alt`, and reads `test_graph_alt` back. The test runs on the destination deployment because the source deployment serves only `test_graph`, and task 2.4, which adds the destination fixture, merges before this group. -- [ ] 3.4 Add `description` to `CommitRequest`, and a repo update that sends `PATCH /api/v1/repos/{owner}/{repo}` with `description`, `readme`, `tags` and `is_public`, to `sdk/src/prompts.rs`; verify with httpmock tests of the request bodies and an integration test that pushes a commit with a description, updates the repo's description, readme, tags and visibility, and reads each back +- [ ] 3.4 Add `description` to `CommitRequest`, `readme` and the repo's `tags` to `Prompt`, which `PromptClient::get` returns and export reads the repo details from, and a repo update that sends `PATCH /api/v1/repos/{owner}/{repo}` with `description`, `readme`, `tags` and `is_public`, to `sdk/src/prompts.rs`; verify with httpmock tests of the request bodies and of a `PromptClient::get` response that holds `readme` and `tags`, and an integration test that pushes a commit with a description, updates the repo's description, readme, tags and visibility, and reads each back through `PromptClient::get` ## 4. Shared code for the prompt and assistant commands: the export file, the comparison and the report From c02e2cd62096c4903ebc288ef01d72a03e2a9a7b Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Tue, 13 Oct 2026 22:53:20 -0400 Subject: [PATCH 34/90] =?UTF-8?q?=F0=9F=93=9A=20docs(openspec):=20list=20P?= =?UTF-8?q?rompt=20among=20group=203's=20SDK=20structs,=20and=20limit=20id?= =?UTF-8?q?entical=20exports=20to=20an=20unchanged=20repo=20and=20tags?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Task 3.4 adds readme and tags to Prompt, which test_prompt_serialization builds with a struct literal, so group 3's preamble lists Prompt. The repo details and the tags that point at a commit change without a new commit, so the identical-export requirement now holds only while they stay unchanged. #807 Export, import, compare, and verify prompts and assistants across LangSmith environments Co-Authored-By: Claude Opus 5.5 --- .../export-import-prompts-and-assistants/specs/prompt/spec.md | 3 ++- openspec/changes/export-import-prompts-and-assistants/tasks.md | 2 +- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md index 71316235d..ad2b4d6b6 100644 --- a/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md +++ b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md @@ -42,11 +42,12 @@ ### Requirement: Write the same file for the same commit -Exporting the same commit twice SHALL produce identical files. The file SHALL hold no export time, and langstar SHALL write object keys in a fixed order. +Exporting the same commit twice SHALL produce identical files when nobody changed the repo details or the tags that point at the commit in between, since those change without a new commit. The file SHALL hold no export time, and langstar SHALL write object keys in a fixed order. #### Scenario: Export twice - **WHEN** a user exports the same commit twice to two files +- **AND** nobody changed the repo details or the commit's tags between the two exports - **THEN** the two files are identical ### Requirement: List the credentials a prompt needs without their values diff --git a/openspec/changes/export-import-prompts-and-assistants/tasks.md b/openspec/changes/export-import-prompts-and-assistants/tasks.md index d51b0237f..30cfdc9ac 100644 --- a/openspec/changes/export-import-prompts-and-assistants/tasks.md +++ b/openspec/changes/export-import-prompts-and-assistants/tasks.md @@ -16,7 +16,7 @@ Each numbered group below is one pull request, merged in the order listed, excep ## 3. SDK reads and writes the fields export and import need -Each task in this group adds a public field to one of these SDK structs: `CommitManifestResponse` (the struct `PromptClient::get_commit` returns), `CommitData`, `CommitRequest`, `Assistant`, `CreateAssistantRequest`, `UpdateAssistantRequest` and `AssistantSearchRequest`. A crate that builds one of them with a struct literal stops compiling, so each task updates every struct literal of its struct in this repository. Whether this group's pull request and task 2.2's, which adds `url` to `Deployment`, carry a `BREAKING CHANGE:` footer waits on [Should the SDK field additions count as a breaking change?](design.md#9---should-the-sdk-field-additions-count-as-a-breaking-change). +Each task in this group adds a public field to one of these SDK structs: `Prompt` (the struct `PromptClient::get` returns, built with a struct literal in `test_prompt_serialization` in `sdk/src/prompts.rs`), `CommitManifestResponse` (the struct `PromptClient::get_commit` returns), `CommitData`, `CommitRequest`, `Assistant`, `CreateAssistantRequest`, `UpdateAssistantRequest` and `AssistantSearchRequest`. A crate that builds one of them with a struct literal stops compiling, so each task updates every struct literal of its struct in this repository. Whether this group's pull request and task 2.2's, which adds `url` to `Deployment`, carry a `BREAKING CHANGE:` footer waits on [Should the SDK field additions count as a breaking change?](design.md#9---should-the-sdk-field-additions-count-as-a-breaking-change). - [ ] 3.1 Add `description`, `model_config` and `model_provider` to `CommitManifestResponse`, and a commit read that sends `include=model` and returns that struct, to `sdk/src/prompts.rs`; verify with an httpmock test that asserts the query string and the parsed fields, and an integration test that reads a prompt with a model - [ ] 3.2 Add the commit's `id` to `CommitData` in `sdk/src/prompts.rs`, a call that lists a prompt's commits with their `id` and hash (`GET /api/v1/commits/{owner}/{repo}`), calls that list, create and move commit tags, which name their commit by that `id` in `POST /api/v1/repos/{owner}/{repo}/tags` and `PATCH /api/v1/repos/{owner}/{repo}/tags/{tag_name}`, and a call that deletes a tag with `DELETE /api/v1/repos/{owner}/{repo}/tags/{tag_name}`, which task 10.3 uses; verify with httpmock tests that assert `commit_id` in each create and move request body and the path of the delete request, and an integration test that pushes two commits, tags the first, finds the second commit's `id` through the commit list, moves the tag to it, reads the tag back through the SDK, deletes it, and asserts that the tag list no longer holds it From 74f815b132bc3f0857a245636440c62287229281 Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Tue, 13 Oct 2026 22:57:49 -0400 Subject: [PATCH 35/90] =?UTF-8?q?=F0=9F=93=9A=20docs(openspec):=20name=20e?= =?UTF-8?q?xhaustive=20destructuring=20alongside=20struct=20literals=20as?= =?UTF-8?q?=20what=20a=20new=20SDK=20field=20breaks?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A new public field also breaks a crate that destructures the struct with a pattern naming every field. The proposal, the design's risk and question 9's context, and group 3's preamble now say so; question 9's recommendation is unchanged and stays with codekiln. #807 Export, import, compare, and verify prompts and assistants across LangSmith environments Co-Authored-By: Claude Opus 5.5 --- .../changes/export-import-prompts-and-assistants/design.md | 4 ++-- .../changes/export-import-prompts-and-assistants/proposal.md | 2 +- .../changes/export-import-prompts-and-assistants/tasks.md | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/openspec/changes/export-import-prompts-and-assistants/design.md b/openspec/changes/export-import-prompts-and-assistants/design.md index f5921e6ef..cf277998e 100644 --- a/openspec/changes/export-import-prompts-and-assistants/design.md +++ b/openspec/changes/export-import-prompts-and-assistants/design.md @@ -110,7 +110,7 @@ A mise task runs a short Python script that pulls the imported prompt with `incl - [LangSmith may store a pushed model differently from the source] → Import reads the prompt back and reports any difference, and in [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8) a developer checks the round trip against the live API before anyone builds the commands. - [Pushing a manifest identical to the latest commit may return 409] → Import compares first and pushes nothing when the prompt matches. - [langstar finds a deployment's Agent Server URL only in `source_config.custom_url` (`resolve_deployment_url` in `cli/src/deployment_utils.rs`), and a deployment built from GitHub has none, so every `langstar assistant` command, the new ones included, fails on such a deployment] → In [export-import-prompts-and-assistants Task 2.2 Return the URL of a deployment built from GitHub](tasks.md#L13), a developer adds the control plane's top-level `url` to the SDK's `Deployment`, which drops it today, and makes `resolve_deployment_url` use it, before anyone builds the assistant export, compare, verify and import commands. -- [The SDK changes for export and import add public fields to `Deployment`, `CommitManifestResponse`, `CommitData`, `CommitRequest`, `Assistant` and the assistant request structs, so a crate that builds one of those structs with a struct literal stops compiling] → Each pull request that adds a field updates every struct literal of that struct in this repository. [Should the SDK field additions count as a breaking change?](#9---should-the-sdk-field-additions-count-as-a-breaking-change) asks codekiln whether those pull requests also carry a `BREAKING CHANGE:` footer, which `docs/dev/git-scm-conventions.md` says makes the release a major version. +- [The SDK changes for export and import add public fields to `Deployment`, `Prompt`, `CommitManifestResponse`, `CommitData`, `CommitRequest`, `Assistant` and the assistant request structs, so a crate that builds one of those structs with a struct literal or destructures it with a pattern that names every field stops compiling] → Each pull request that adds a field updates every struct literal of that struct in this repository. [Should the SDK field additions count as a breaking change?](#9---should-the-sdk-field-additions-count-as-a-breaking-change) asks codekiln whether those pull requests also carry a `BREAKING CHANGE:` footer, which `docs/dev/git-scm-conventions.md` says makes the release a major version. - [A tag in the destination may already point at another commit] → `--update` moves the tag, and `--dry-run` lists the move. - [The control plane may refuse a second deployment from this repository with 409] → [export-import-prompts-and-assistants Task 2.3 Register the test graph under a second graph ID](tasks.md#L14) tries one before anyone builds the fixture, and a 409 sends [Which second workspace and second deployment should the tests use?](#5---which-second-workspace-and-second-deployment-should-the-tests-use) back to codekiln. @@ -179,6 +179,6 @@ None. ### 9 - Should the SDK field additions count as a breaking change? > Context from the drafting agent, for question 9. -> A new public field breaks only a crate that builds the struct with a struct literal. The only crate known to use the SDK is langstar's own CLI, and each pull request that adds a field updates the CLI's struct literals. `sdk/Cargo.toml` leaves the crate publishable, though, so an outside crate could break, and `docs/dev/git-scm-conventions.md` gives every API break a `BREAKING CHANGE:` footer, which makes the release a major version. The structs that gain fields are `Deployment`, `Prompt` (which `PromptClient::get` returns), `CommitManifestResponse` (which `PromptClient::get_commit` returns), `CommitData`, `CommitRequest`, `Assistant` and the assistant request structs. Marking them `#[non_exhaustive]` would stop every crate except the SDK itself from building them with struct literals, even with `..Default::default()`. langstar's own CLI builds `Assistant`, `CommitRequest`, `CreateAssistantRequest`, `UpdateAssistantRequest` and `Deployment` with struct literals today, and the SDK's integration tests build all of them except `Assistant` the same way, so the SDK would need a constructor for each of those five structs before the CLI and those tests compile again. After that one break, a new field would break nothing. [🚨 BREAKING CHANGE(sdk): move annotation-queue run calls to the platform items API · Pull Request #815](https://github.com/codekiln/langstar/pull/815) removes SDK functions and changes return types, marks itself breaking, and leaves that choice to codekiln too, so one answer to this question can set the policy for both pull requests. Recommendation: treat the additions as `feat` changes with no `BREAKING CHANGE:` footer, and name the new fields in the release notes. +> A new public field breaks only a crate that builds the struct with a struct literal or destructures it with a pattern that names every field; nothing in this repository destructures these structs that way. The only crate known to use the SDK is langstar's own CLI, and each pull request that adds a field updates the CLI's struct literals. `sdk/Cargo.toml` leaves the crate publishable, though, so an outside crate could break, and `docs/dev/git-scm-conventions.md` gives every API break a `BREAKING CHANGE:` footer, which makes the release a major version. The structs that gain fields are `Deployment`, `Prompt` (which `PromptClient::get` returns), `CommitManifestResponse` (which `PromptClient::get_commit` returns), `CommitData`, `CommitRequest`, `Assistant` and the assistant request structs. Marking them `#[non_exhaustive]` would stop every crate except the SDK itself from building them with struct literals, even with `..Default::default()`. langstar's own CLI builds `Assistant`, `CommitRequest`, `CreateAssistantRequest`, `UpdateAssistantRequest` and `Deployment` with struct literals today, and the SDK's integration tests build all of them except `Assistant` the same way, so the SDK would need a constructor for each of those five structs before the CLI and those tests compile again. After that one break, a new field would break nothing. [🚨 BREAKING CHANGE(sdk): move annotation-queue run calls to the platform items API · Pull Request #815](https://github.com/codekiln/langstar/pull/815) removes SDK functions and changes return types, marks itself breaking, and leaves that choice to codekiln too, so one answer to this question can set the policy for both pull requests. Recommendation: treat the additions as `feat` changes with no `BREAKING CHANGE:` footer, and name the new fields in the release notes. diff --git a/openspec/changes/export-import-prompts-and-assistants/proposal.md b/openspec/changes/export-import-prompts-and-assistants/proposal.md index d165ab2cf..86df88fa2 100644 --- a/openspec/changes/export-import-prompts-and-assistants/proposal.md +++ b/openspec/changes/export-import-prompts-and-assistants/proposal.md @@ -30,7 +30,7 @@ A prompt belongs to a workspace and an assistant belongs to a deployment, but bo ## Impact -- SDK: `sdk/src/prompts.rs` gains a commit read that asks for the model and returns it on `CommitManifestResponse`, a commit description on commit create, the new commit's `id`, the repo's readme and tags on `Prompt`, a repo update, a commit list that returns each commit's `id`, calls that list, create and move commit tags, and a tag delete that the end-to-end test uses. `sdk/src/assistants.rs` gains the `context` and `version` fields, `graph_id` on update, and a search by graph ID. `sdk/src/deployments.rs` gains the deployment's `url`. Each new public field breaks a crate outside this repository that builds that SDK struct with a struct literal; codekiln decides in [Should the SDK field additions count as a breaking change?](design.md#9---should-the-sdk-field-additions-count-as-a-breaking-change) whether that makes the release a major version. +- SDK: `sdk/src/prompts.rs` gains a commit read that asks for the model and returns it on `CommitManifestResponse`, a commit description on commit create, the new commit's `id`, the repo's readme and tags on `Prompt`, a repo update, a commit list that returns each commit's `id`, calls that list, create and move commit tags, and a tag delete that the end-to-end test uses. `sdk/src/assistants.rs` gains the `context` and `version` fields, `graph_id` on update, and a search by graph ID. `sdk/src/deployments.rs` gains the deployment's `url`. Each new public field breaks a crate outside this repository that builds that SDK struct with a struct literal or destructures it with a pattern that names every field; codekiln decides in [Should the SDK field additions count as a breaking change?](design.md#9---should-the-sdk-field-additions-count-as-a-breaking-change) whether that makes the release a major version. - CLI: new subcommands in `cli/src/commands/prompt.rs` and `cli/src/commands/assistant.rs`, and a shared module for the export file, the destination record, comparison and the exit status. `resolve_deployment_url` in `cli/src/deployment_utils.rs` also returns the Agent Server URL of a deployment built from GitHub, which every `langstar assistant` command needs. - Tests need a second workspace, a second deployment and one model provider key for the structured output check. codekiln creates the workspace and the key, and langstar's tests read them through `fnox.toml` and the CI workflow; the destination deployment fixture from [export-import-prompts-and-assistants Task 2.4 Add a destination deployment fixture](tasks.md#L15) creates the deployment whenever the cleanup workflow has deleted it, as the source fixture does today. Nobody knows yet whether the control plane accepts a second deployment from this repository, and [export-import-prompts-and-assistants Task 2.3 Register the test graph under a second graph ID](tasks.md#L14) finds out before anyone builds the fixture. - Docs: a new usage page on moving prompts and assistants between environments. diff --git a/openspec/changes/export-import-prompts-and-assistants/tasks.md b/openspec/changes/export-import-prompts-and-assistants/tasks.md index 30cfdc9ac..577793e3b 100644 --- a/openspec/changes/export-import-prompts-and-assistants/tasks.md +++ b/openspec/changes/export-import-prompts-and-assistants/tasks.md @@ -16,7 +16,7 @@ Each numbered group below is one pull request, merged in the order listed, excep ## 3. SDK reads and writes the fields export and import need -Each task in this group adds a public field to one of these SDK structs: `Prompt` (the struct `PromptClient::get` returns, built with a struct literal in `test_prompt_serialization` in `sdk/src/prompts.rs`), `CommitManifestResponse` (the struct `PromptClient::get_commit` returns), `CommitData`, `CommitRequest`, `Assistant`, `CreateAssistantRequest`, `UpdateAssistantRequest` and `AssistantSearchRequest`. A crate that builds one of them with a struct literal stops compiling, so each task updates every struct literal of its struct in this repository. Whether this group's pull request and task 2.2's, which adds `url` to `Deployment`, carry a `BREAKING CHANGE:` footer waits on [Should the SDK field additions count as a breaking change?](design.md#9---should-the-sdk-field-additions-count-as-a-breaking-change). +Each task in this group adds a public field to one of these SDK structs: `Prompt` (the struct `PromptClient::get` returns, built with a struct literal in `test_prompt_serialization` in `sdk/src/prompts.rs`), `CommitManifestResponse` (the struct `PromptClient::get_commit` returns), `CommitData`, `CommitRequest`, `Assistant`, `CreateAssistantRequest`, `UpdateAssistantRequest` and `AssistantSearchRequest`. A crate that builds one of them with a struct literal, or destructures one with a pattern that names every field, stops compiling; nothing in this repository destructures them that way, so each task updates every struct literal of its struct in this repository. Whether this group's pull request and task 2.2's, which adds `url` to `Deployment`, carry a `BREAKING CHANGE:` footer waits on [Should the SDK field additions count as a breaking change?](design.md#9---should-the-sdk-field-additions-count-as-a-breaking-change). - [ ] 3.1 Add `description`, `model_config` and `model_provider` to `CommitManifestResponse`, and a commit read that sends `include=model` and returns that struct, to `sdk/src/prompts.rs`; verify with an httpmock test that asserts the query string and the parsed fields, and an integration test that reads a prompt with a model - [ ] 3.2 Add the commit's `id` to `CommitData` in `sdk/src/prompts.rs`, a call that lists a prompt's commits with their `id` and hash (`GET /api/v1/commits/{owner}/{repo}`), calls that list, create and move commit tags, which name their commit by that `id` in `POST /api/v1/repos/{owner}/{repo}/tags` and `PATCH /api/v1/repos/{owner}/{repo}/tags/{tag_name}`, and a call that deletes a tag with `DELETE /api/v1/repos/{owner}/{repo}/tags/{tag_name}`, which task 10.3 uses; verify with httpmock tests that assert `commit_id` in each create and move request body and the path of the delete request, and an integration test that pushes two commits, tags the first, finds the second commit's `id` through the commit list, moves the tag to it, reads the tag back through the SDK, deletes it, and asserts that the tag list no longer holds it From d8ae4cba7203753af75feee7be0bbbf34a72aa79 Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Tue, 13 Oct 2026 23:10:32 -0400 Subject: [PATCH 36/90] =?UTF-8?q?=F0=9F=93=9A=20docs(openspec):=20give=20t?= =?UTF-8?q?he=20409=20check=20a=20developer=20as=20subject,=20split=20task?= =?UTF-8?q?=205.1=20into=20its=20tests,=20and=20drop=20the=20import-skips-?= =?UTF-8?q?comparison=20negation?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Name a developer as who deploys in task 2.3, read the commit description through the commit read from task 3.1, fix the print_message permalink line, and untangle "calls that list". #807 Export, import, compare, and verify prompts and assistants across LangSmith environments Co-Authored-By: Claude Opus 5.5 --- .../export-import-prompts-and-assistants/design.md | 8 ++++---- .../export-import-prompts-and-assistants/proposal.md | 4 ++-- .../specs/prompt/spec.md | 2 +- .../export-import-prompts-and-assistants/tasks.md | 10 +++++++--- 4 files changed, 14 insertions(+), 10 deletions(-) diff --git a/openspec/changes/export-import-prompts-and-assistants/design.md b/openspec/changes/export-import-prompts-and-assistants/design.md index cf277998e..baafe0bc7 100644 --- a/openspec/changes/export-import-prompts-and-assistants/design.md +++ b/openspec/changes/export-import-prompts-and-assistants/design.md @@ -51,7 +51,7 @@ An assistant export has `kind: "assistant"`, a `source` with the deployment, ass The file holds no export time and langstar sorts its keys, so exporting an unchanged resource leaves Git with nothing to commit. -Without `--output`, export prints the file to stdout and nothing else, so `langstar prompt export my-prompt > my-prompt.json` writes a file import can read. Export writes its progress messages to stderr in every output format. [`OutputFormatter::print_message` in cli/src/output.rs:172](https://github.com/codekiln/langstar/blob/e4ae746d51f1f3f9ccf3fc97787148c3b4df2bce/cli/src/output.rs#L172) sends `info` to stdout in table mode, so export does not use it for those messages. +Without `--output`, export prints the file to stdout and nothing else, so `langstar prompt export my-prompt > my-prompt.json` writes a file import can read. Export writes its progress messages to stderr in every output format. [`OutputFormatter::print_message` in cli/src/output.rs:173](https://github.com/codekiln/langstar/blob/e4ae746d51f1f3f9ccf3fc97787148c3b4df2bce/cli/src/output.rs#L173), which `info` calls, prints to stdout in table mode, so export does not use it for those messages. Import, compare and verify read a file only when its `format_version` is 1 and its `kind` matches the command: `prompt` for the prompt commands and `assistant` for the assistant commands. For any other file the command writes nothing, exits with status 1 and names the field and the value it found. When a user runs `prompt import` on an assistant export, or on a file a newer langstar wrote, langstar stops with that error before it changes anything in the destination. @@ -87,7 +87,7 @@ Assistant verify checks the graph with `POST /assistants/search` filtered to the ### Import checks the destination's secrets, graph and prompts before it writes -Before it writes anything, import checks that the destination has the secrets, the graph and the prompts the copy needs. These are the checks `verify` makes, and import runs them because [#807 Export, import, compare, and verify prompts and assistants across LangSmith environments](https://github.com/codekiln/langstar/issues/807) asks import to check the destination's dependencies. Import skips verify's comparison, since the destination may have no copy yet. Prompt import stops when the destination workspace lacks a secret in `requires.secrets`, and assistant import stops when the deployment does not serve the file's graph or the workspace, from `--workspace-id` or the configured workspace, lacks a prompt in `requires.prompts`. Each names what is missing, writes nothing and exits with status 3, and `--dry-run` reports the same failed checks. A user therefore creates the destination workspace's secrets before importing a prompt, and imports the prompts before importing an assistant that refers to them. [Should a failed dependency check stop import before it writes?](#8---should-a-failed-dependency-check-stop-import-before-it-writes) asks whether import should write anyway and report the failed checks afterwards. +Before it writes anything, import checks that the destination has the secrets, the graph and the prompts the copy needs. These are the checks `verify` makes, and import runs them because [#807 Export, import, compare, and verify prompts and assistants across LangSmith environments](https://github.com/codekiln/langstar/issues/807) asks import to check the destination's dependencies. Prompt import stops when the destination workspace lacks a secret in `requires.secrets`, and assistant import stops when the deployment does not serve the file's graph or the workspace, from `--workspace-id` or the configured workspace, lacks a prompt in `requires.prompts`. Each names what is missing, writes nothing and exits with status 3, and `--dry-run` reports the same failed checks. A user therefore creates the destination workspace's secrets before importing a prompt, and imports the prompts before importing an assistant that refers to them. [Should a failed dependency check stop import before it writes?](#8---should-a-failed-dependency-check-stop-import-before-it-writes) asks whether import should write anyway and report the failed checks afterwards. ### The user names an assistant's prompt references @@ -112,7 +112,7 @@ A mise task runs a short Python script that pulls the imported prompt with `incl - [langstar finds a deployment's Agent Server URL only in `source_config.custom_url` (`resolve_deployment_url` in `cli/src/deployment_utils.rs`), and a deployment built from GitHub has none, so every `langstar assistant` command, the new ones included, fails on such a deployment] → In [export-import-prompts-and-assistants Task 2.2 Return the URL of a deployment built from GitHub](tasks.md#L13), a developer adds the control plane's top-level `url` to the SDK's `Deployment`, which drops it today, and makes `resolve_deployment_url` use it, before anyone builds the assistant export, compare, verify and import commands. - [The SDK changes for export and import add public fields to `Deployment`, `Prompt`, `CommitManifestResponse`, `CommitData`, `CommitRequest`, `Assistant` and the assistant request structs, so a crate that builds one of those structs with a struct literal or destructures it with a pattern that names every field stops compiling] → Each pull request that adds a field updates every struct literal of that struct in this repository. [Should the SDK field additions count as a breaking change?](#9---should-the-sdk-field-additions-count-as-a-breaking-change) asks codekiln whether those pull requests also carry a `BREAKING CHANGE:` footer, which `docs/dev/git-scm-conventions.md` says makes the release a major version. - [A tag in the destination may already point at another commit] → `--update` moves the tag, and `--dry-run` lists the move. -- [The control plane may refuse a second deployment from this repository with 409] → [export-import-prompts-and-assistants Task 2.3 Register the test graph under a second graph ID](tasks.md#L14) tries one before anyone builds the fixture, and a 409 sends [Which second workspace and second deployment should the tests use?](#5---which-second-workspace-and-second-deployment-should-the-tests-use) back to codekiln. +- [The control plane may refuse a second deployment from this repository with 409] → In [export-import-prompts-and-assistants Task 2.3 Register the test graph under a second graph ID](tasks.md#L14), a developer deploys the new `langgraph.json` once, before anyone builds the fixture. If the control plane answers with 409, the developer brings [Which second workspace and second deployment should the tests use?](#5---which-second-workspace-and-second-deployment-should-the-tests-use) back to codekiln. ## Resolved Questions @@ -151,7 +151,7 @@ None. ### 5 - Which second workspace and second deployment should the tests use? > Context from the drafting agent, for question 5. -> The import tests need a workspace and a deployment to import into. The draft has codekiln create a second workspace in the test organization, which the existing API key can reach, and the tests read its ID from `LANGSTAR_TEST_DEST_WORKSPACE_ID`. The destination deployment fixture from [export-import-prompts-and-assistants Task 2.4 Add a destination deployment fixture](tasks.md#L15) deploys the test graph a second time from a `langgraph.json` of its own, which registers the graph as both `test_graph` and `test_graph_alt` so the update tests can move an assistant from one graph to the other. Nobody knows yet whether the control plane accepts that deployment. It answers a second deployment of the same agent environment with 409 "A deployment already exists for this agent environment", and no document says whether a second `langgraph.json` in the same repository makes a new agent environment. The check in [export-import-prompts-and-assistants Task 2.3 Register the test graph under a second graph ID](tasks.md#L14) is the first deployment that tries. If it gets the 409, the developer stops before task 2.4 and brings this question back to codekiln with what the control plane returned. `.github/workflows/cleanup-test-deployments.yml` deletes each test deployment once it is four hours old and the fixture rebuilds it on the next CI run, so the first CI run after each cleanup waits for two deployments to build instead of one. Recommendation: a second workspace in the same organization, which codekiln creates, and one destination deployment that the fixture from task 2.4 rebuilds whenever the cleanup workflow has deleted it. +> The import tests need a workspace and a deployment to import into. The draft has codekiln create a second workspace in the test organization, which the existing API key can reach, and the tests read its ID from `LANGSTAR_TEST_DEST_WORKSPACE_ID`. The destination deployment fixture from [export-import-prompts-and-assistants Task 2.4 Add a destination deployment fixture](tasks.md#L15) deploys the test graph a second time from a `langgraph.json` of its own, which registers the graph as both `test_graph` and `test_graph_alt` so the update tests can move an assistant from one graph to the other. Nobody knows yet whether the control plane accepts that deployment. It answers a second deployment of the same agent environment with 409 "A deployment already exists for this agent environment", and no document says whether a second `langgraph.json` in the same repository makes a new agent environment. In [export-import-prompts-and-assistants Task 2.3 Register the test graph under a second graph ID](tasks.md#L14), a developer deploys the new `langgraph.json` once to find out. If the control plane answers with the 409, the developer stops before task 2.4 and brings this question back to codekiln with what the control plane returned. `.github/workflows/cleanup-test-deployments.yml` deletes each test deployment once it is four hours old and the fixture rebuilds it on the next CI run, so the first CI run after each cleanup waits for two deployments to build instead of one. Recommendation: a second workspace in the same organization, which codekiln creates, and one destination deployment that the fixture from task 2.4 rebuilds whenever the cleanup workflow has deleted it. diff --git a/openspec/changes/export-import-prompts-and-assistants/proposal.md b/openspec/changes/export-import-prompts-and-assistants/proposal.md index 86df88fa2..351b06c34 100644 --- a/openspec/changes/export-import-prompts-and-assistants/proposal.md +++ b/openspec/changes/export-import-prompts-and-assistants/proposal.md @@ -30,9 +30,9 @@ A prompt belongs to a workspace and an assistant belongs to a deployment, but bo ## Impact -- SDK: `sdk/src/prompts.rs` gains a commit read that asks for the model and returns it on `CommitManifestResponse`, a commit description on commit create, the new commit's `id`, the repo's readme and tags on `Prompt`, a repo update, a commit list that returns each commit's `id`, calls that list, create and move commit tags, and a tag delete that the end-to-end test uses. `sdk/src/assistants.rs` gains the `context` and `version` fields, `graph_id` on update, and a search by graph ID. `sdk/src/deployments.rs` gains the deployment's `url`. Each new public field breaks a crate outside this repository that builds that SDK struct with a struct literal or destructures it with a pattern that names every field; codekiln decides in [Should the SDK field additions count as a breaking change?](design.md#9---should-the-sdk-field-additions-count-as-a-breaking-change) whether that makes the release a major version. +- SDK: `sdk/src/prompts.rs` gains a commit read that asks for the model and returns it on `CommitManifestResponse`, a commit description on commit create, the new commit's `id`, the repo's readme and tags on `Prompt`, a repo update, a commit list that returns each commit's `id`, calls to list, create and move commit tags, and a tag delete that the end-to-end test uses. `sdk/src/assistants.rs` gains the `context` and `version` fields, `graph_id` on update, and a search by graph ID. `sdk/src/deployments.rs` gains the deployment's `url`. Each new public field breaks a crate outside this repository that builds that SDK struct with a struct literal or destructures it with a pattern that names every field; codekiln decides in [Should the SDK field additions count as a breaking change?](design.md#9---should-the-sdk-field-additions-count-as-a-breaking-change) whether that makes the release a major version. - CLI: new subcommands in `cli/src/commands/prompt.rs` and `cli/src/commands/assistant.rs`, and a shared module for the export file, the destination record, comparison and the exit status. `resolve_deployment_url` in `cli/src/deployment_utils.rs` also returns the Agent Server URL of a deployment built from GitHub, which every `langstar assistant` command needs. -- Tests need a second workspace, a second deployment and one model provider key for the structured output check. codekiln creates the workspace and the key, and langstar's tests read them through `fnox.toml` and the CI workflow; the destination deployment fixture from [export-import-prompts-and-assistants Task 2.4 Add a destination deployment fixture](tasks.md#L15) creates the deployment whenever the cleanup workflow has deleted it, as the source fixture does today. Nobody knows yet whether the control plane accepts a second deployment from this repository, and [export-import-prompts-and-assistants Task 2.3 Register the test graph under a second graph ID](tasks.md#L14) finds out before anyone builds the fixture. +- Tests need a second workspace, a second deployment and one model provider key for the structured output check. codekiln creates the workspace and the key, and langstar's tests read them through `fnox.toml` and the CI workflow; the destination deployment fixture from [export-import-prompts-and-assistants Task 2.4 Add a destination deployment fixture](tasks.md#L15) creates the deployment whenever the cleanup workflow has deleted it, as the source fixture does today. Nobody knows yet whether the control plane accepts a second deployment from this repository. In [export-import-prompts-and-assistants Task 2.3 Register the test graph under a second graph ID](tasks.md#L14), a developer deploys the new `langgraph.json` once to find out, before anyone builds the fixture. - Docs: a new usage page on moving prompts and assistants between environments. ## Citations diff --git a/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md index ad2b4d6b6..58ad3a034 100644 --- a/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md +++ b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md @@ -89,7 +89,7 @@ When the manifest holds no chat model constructor, `langstar prompt export` SHAL ### Requirement: Check the destination before importing a prompt -Before it writes, `langstar prompt import` SHALL run the secret check and the saved model configuration check of `prompt verify` against the destination workspace, and not its comparison, because the destination may have no prompt yet. When the workspace lacks a secret in `requires.secrets` it SHALL write nothing, name each missing secret and exit with status 3. A missing saved model configuration SHALL be a note, as in `prompt verify`. +Before it writes, `langstar prompt import` SHALL run the secret check and the saved model configuration check of `prompt verify` against the destination workspace. When the workspace lacks a secret in `requires.secrets` it SHALL write nothing, name each missing secret and exit with status 3. A missing saved model configuration SHALL be a note, as in `prompt verify`. #### Scenario: Secret missing at import diff --git a/openspec/changes/export-import-prompts-and-assistants/tasks.md b/openspec/changes/export-import-prompts-and-assistants/tasks.md index 577793e3b..2dc53b895 100644 --- a/openspec/changes/export-import-prompts-and-assistants/tasks.md +++ b/openspec/changes/export-import-prompts-and-assistants/tasks.md @@ -19,9 +19,9 @@ Each numbered group below is one pull request, merged in the order listed, excep Each task in this group adds a public field to one of these SDK structs: `Prompt` (the struct `PromptClient::get` returns, built with a struct literal in `test_prompt_serialization` in `sdk/src/prompts.rs`), `CommitManifestResponse` (the struct `PromptClient::get_commit` returns), `CommitData`, `CommitRequest`, `Assistant`, `CreateAssistantRequest`, `UpdateAssistantRequest` and `AssistantSearchRequest`. A crate that builds one of them with a struct literal, or destructures one with a pattern that names every field, stops compiling; nothing in this repository destructures them that way, so each task updates every struct literal of its struct in this repository. Whether this group's pull request and task 2.2's, which adds `url` to `Deployment`, carry a `BREAKING CHANGE:` footer waits on [Should the SDK field additions count as a breaking change?](design.md#9---should-the-sdk-field-additions-count-as-a-breaking-change). - [ ] 3.1 Add `description`, `model_config` and `model_provider` to `CommitManifestResponse`, and a commit read that sends `include=model` and returns that struct, to `sdk/src/prompts.rs`; verify with an httpmock test that asserts the query string and the parsed fields, and an integration test that reads a prompt with a model -- [ ] 3.2 Add the commit's `id` to `CommitData` in `sdk/src/prompts.rs`, a call that lists a prompt's commits with their `id` and hash (`GET /api/v1/commits/{owner}/{repo}`), calls that list, create and move commit tags, which name their commit by that `id` in `POST /api/v1/repos/{owner}/{repo}/tags` and `PATCH /api/v1/repos/{owner}/{repo}/tags/{tag_name}`, and a call that deletes a tag with `DELETE /api/v1/repos/{owner}/{repo}/tags/{tag_name}`, which task 10.3 uses; verify with httpmock tests that assert `commit_id` in each create and move request body and the path of the delete request, and an integration test that pushes two commits, tags the first, finds the second commit's `id` through the commit list, moves the tag to it, reads the tag back through the SDK, deletes it, and asserts that the tag list no longer holds it +- [ ] 3.2 Add the commit's `id` to `CommitData` in `sdk/src/prompts.rs`, a call that lists a prompt's commits with their `id` and hash (`GET /api/v1/commits/{owner}/{repo}`), calls to list, create and move commit tags, which name their commit by that `id` in `POST /api/v1/repos/{owner}/{repo}/tags` and `PATCH /api/v1/repos/{owner}/{repo}/tags/{tag_name}`, and a call that deletes a tag with `DELETE /api/v1/repos/{owner}/{repo}/tags/{tag_name}`, which task 10.3 uses; verify with httpmock tests that assert `commit_id` in each create and move request body and the path of the delete request, and an integration test that pushes two commits, tags the first, finds the second commit's `id` through the commit list, moves the tag to it, reads the tag back through the SDK, deletes it, and asserts that the tag list no longer holds it - [ ] 3.3 Add `context` and `version` to `Assistant`, `context` and `graph_id` to `UpdateAssistantRequest`, `context` to `CreateAssistantRequest`, and a `graph_id` filter to `AssistantSearchRequest`, in `sdk/src/assistants.rs`; verify with httpmock tests of the request bodies and an integration test that creates an assistant with context on the destination deployment from 2.4, reads it back with its version, finds it by graph ID, moves it from `test_graph` to `test_graph_alt`, and reads `test_graph_alt` back. The test runs on the destination deployment because the source deployment serves only `test_graph`, and task 2.4, which adds the destination fixture, merges before this group. -- [ ] 3.4 Add `description` to `CommitRequest`, `readme` and the repo's `tags` to `Prompt`, which `PromptClient::get` returns and export reads the repo details from, and a repo update that sends `PATCH /api/v1/repos/{owner}/{repo}` with `description`, `readme`, `tags` and `is_public`, to `sdk/src/prompts.rs`; verify with httpmock tests of the request bodies and of a `PromptClient::get` response that holds `readme` and `tags`, and an integration test that pushes a commit with a description, updates the repo's description, readme, tags and visibility, and reads each back through `PromptClient::get` +- [ ] 3.4 Add `description` to `CommitRequest`, `readme` and the repo's `tags` to `Prompt`, which `PromptClient::get` returns and export reads the repo details from, and a repo update that sends `PATCH /api/v1/repos/{owner}/{repo}` with `description`, `readme`, `tags` and `is_public`, to `sdk/src/prompts.rs`; verify with httpmock tests of the request bodies and of a `PromptClient::get` response that holds `readme` and `tags`, and an integration test that pushes a commit with a description and reads it back through the commit read from 3.1, then updates the repo's description, readme, tags and visibility and reads each back through `PromptClient::get` ## 4. Shared code for the prompt and assistant commands: the export file, the comparison and the report @@ -32,7 +32,11 @@ Each task in this group adds a public field to one of these SDK structs: `Prompt ## 5. prompt export -- [ ] 5.1 Add `langstar prompt export` with `--commit`, `--output` and the scoping flags; verify with integration tests that create prompts through the SDK (a `json_schema` prompt, a `function_calling` prompt with a nested schema, a prompt with several messages and a placeholder, and a prompt with a model and metadata), export each through the CLI, and assert every field of the file against the SDK's read of the commit, an export with `--commit prod` of a prompt whose `prod` tag points at an earlier commit, asserting that `source.commit_hash` and the manifest are that earlier commit's, and an error test that exporting a handle no prompt in the workspace has exits with status 1, names the handle and writes no file, and that the file for the prompt with a model lists `OPENAI_API_KEY` under `requires.secrets` and `ChatOpenAI` under `requires.model_class` and holds no secret value, and that the file for the `json_schema` prompt, which has no model, holds an empty `requires.secrets` and null under `requires.model_class`, and a CLI test that runs `prompt export` without `--output`, parses all of stdout as one JSON document, and asserts that its `source.commit_hash` matches the SDK's read and that any progress message is on stderr +- [ ] 5.1 Add `langstar prompt export` with `--commit`, `--output` and the scoping flags. Verify with these tests: + - Integration tests that create four prompts through the SDK (a `json_schema` prompt with no model, a `function_calling` prompt with a nested schema, a prompt with several messages and a placeholder, and a prompt with a model and metadata), export each through the CLI, and assert every field of the file against the SDK's read of the commit. The file for the prompt with a model lists `OPENAI_API_KEY` under `requires.secrets` and `ChatOpenAI` under `requires.model_class`, and holds no secret value. The file for the `json_schema` prompt holds an empty `requires.secrets` and null under `requires.model_class`. + - An integration test that exports, with `--commit prod`, a prompt whose `prod` tag points at an earlier commit, and asserts that `source.commit_hash` and the manifest are that earlier commit's. + - An error test that exports a handle the workspace has no prompt for, and asserts that it exits with status 1, names the handle and writes no file. + - A CLI test that runs `prompt export` without `--output`, parses all of stdout as one JSON document, and asserts that its `source.commit_hash` matches the SDK's read and that any progress message is on stderr. - [ ] 5.2 Document `prompt export` in `docs/usage/` and the README; verify that each documented command runs as written ## 6. prompt compare and verify From 0d528e3994e159452f412d7ee47d17bcbcfcf828 Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Tue, 13 Oct 2026 23:21:00 -0400 Subject: [PATCH 37/90] =?UTF-8?q?=F0=9F=93=9A=20docs(openspec):=20record?= =?UTF-8?q?=20a=20new=20assistant's=20ID=20as=20pending=20before=20import?= =?UTF-8?q?=20creates=20it=20with=20if=5Fexists=20do=5Fnothing?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Import chooses the assistant ID, writes it to the destination record as pending, and creates the assistant with that assistant_id and if_exists set to do_nothing, so an import that stops between the create and the record write leaves no second assistant on retry. Task 3.3 adds both fields to CreateAssistantRequest and task 10.1 tests the retry. #807 Export, import, compare, and verify prompts and assistants across LangSmith environments Co-Authored-By: Claude Opus 5.5 --- .../export-import-prompts-and-assistants/design.md | 2 +- .../proposal.md | 2 +- .../specs/assistant/spec.md | 14 +++++++++++--- .../export-import-prompts-and-assistants/tasks.md | 4 ++-- 4 files changed, 15 insertions(+), 7 deletions(-) diff --git a/openspec/changes/export-import-prompts-and-assistants/design.md b/openspec/changes/export-import-prompts-and-assistants/design.md index baafe0bc7..d8f201303 100644 --- a/openspec/changes/export-import-prompts-and-assistants/design.md +++ b/openspec/changes/export-import-prompts-and-assistants/design.md @@ -61,7 +61,7 @@ Export reads the commit with `include=model`, the form the Go SDK uses (`referen ### Import records what it created in a separate file -Import writes the destination's IDs to `.destinations.json`, keyed by destination workspace ID for prompts and by deployment ID for assistants. The export file stays the same after any number of imports, and a fresh export never wipes out the record of earlier imports. A repeated import reads the record to find the copy it made. For a prompt, langstar finds the copy by its handle, and compares the commit hash in the record with the destination's latest commit to tell whether someone changed the prompt since the last import. For an assistant, whose ID differs on every deployment, the record is the only link from the file to its copy. Import writes the new commit hash or assistant ID to the record before it reads the copy back, so an import whose read-back finds a difference still records what it wrote. The next prompt import then compares the destination's latest commit with the commit langstar wrote, and reports a change only when someone else pushed one; the next assistant import finds that assistant instead of creating a second one. +Import writes the destination's IDs to `.destinations.json`, keyed by destination workspace ID for prompts and by deployment ID for assistants. The export file stays the same after any number of imports, and a fresh export never wipes out the record of earlier imports. A repeated import reads the record to find the copy it made. For a prompt, langstar finds the copy by its handle, and compares the commit hash in the record with the destination's latest commit to tell whether someone changed the prompt since the last import. For an assistant, whose ID differs on every deployment, the record is the only link from the file to its copy. Import writes the new commit hash or assistant ID to the record before it reads the copy back, so an import whose read-back finds a difference still records what it wrote. The next prompt import then compares the destination's latest commit with the commit langstar wrote, and reports a change only when someone else pushed one; the next assistant import finds that assistant instead of creating a second one. Import also chooses a new assistant's ID itself and writes it to the record, marked pending, before it sends the create request with that `assistant_id` and `if_exists` set to `do_nothing`, which the Agent Server's `AssistantCreate` accepts. If langstar stops between the create and the record write, the next import finds the pending entry and sends the same create again, which returns the assistant the first import made instead of creating a second one. ### Import changes an existing resource only with --update diff --git a/openspec/changes/export-import-prompts-and-assistants/proposal.md b/openspec/changes/export-import-prompts-and-assistants/proposal.md index 351b06c34..3107e6fc1 100644 --- a/openspec/changes/export-import-prompts-and-assistants/proposal.md +++ b/openspec/changes/export-import-prompts-and-assistants/proposal.md @@ -30,7 +30,7 @@ A prompt belongs to a workspace and an assistant belongs to a deployment, but bo ## Impact -- SDK: `sdk/src/prompts.rs` gains a commit read that asks for the model and returns it on `CommitManifestResponse`, a commit description on commit create, the new commit's `id`, the repo's readme and tags on `Prompt`, a repo update, a commit list that returns each commit's `id`, calls to list, create and move commit tags, and a tag delete that the end-to-end test uses. `sdk/src/assistants.rs` gains the `context` and `version` fields, `graph_id` on update, and a search by graph ID. `sdk/src/deployments.rs` gains the deployment's `url`. Each new public field breaks a crate outside this repository that builds that SDK struct with a struct literal or destructures it with a pattern that names every field; codekiln decides in [Should the SDK field additions count as a breaking change?](design.md#9---should-the-sdk-field-additions-count-as-a-breaking-change) whether that makes the release a major version. +- SDK: `sdk/src/prompts.rs` gains a commit read that asks for the model and returns it on `CommitManifestResponse`, a commit description on commit create, the new commit's `id`, the repo's readme and tags on `Prompt`, a repo update, a commit list that returns each commit's `id`, calls to list, create and move commit tags, and a tag delete that the end-to-end test uses. `sdk/src/assistants.rs` gains the `context` and `version` fields, `assistant_id` and `if_exists` on create, `graph_id` on update, and a search by graph ID. `sdk/src/deployments.rs` gains the deployment's `url`. Each new public field breaks a crate outside this repository that builds that SDK struct with a struct literal or destructures it with a pattern that names every field; codekiln decides in [Should the SDK field additions count as a breaking change?](design.md#9---should-the-sdk-field-additions-count-as-a-breaking-change) whether that makes the release a major version. - CLI: new subcommands in `cli/src/commands/prompt.rs` and `cli/src/commands/assistant.rs`, and a shared module for the export file, the destination record, comparison and the exit status. `resolve_deployment_url` in `cli/src/deployment_utils.rs` also returns the Agent Server URL of a deployment built from GitHub, which every `langstar assistant` command needs. - Tests need a second workspace, a second deployment and one model provider key for the structured output check. codekiln creates the workspace and the key, and langstar's tests read them through `fnox.toml` and the CI workflow; the destination deployment fixture from [export-import-prompts-and-assistants Task 2.4 Add a destination deployment fixture](tasks.md#L15) creates the deployment whenever the cleanup workflow has deleted it, as the source fixture does today. Nobody knows yet whether the control plane accepts a second deployment from this repository. In [export-import-prompts-and-assistants Task 2.3 Register the test graph under a second graph ID](tasks.md#L14), a developer deploys the new `langgraph.json` once to find out, before anyone builds the fixture. - Docs: a new usage page on moving prompts and assistants between environments. diff --git a/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md b/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md index d0ef37e6c..78c6ab744 100644 --- a/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md +++ b/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md @@ -66,14 +66,22 @@ Every `langstar assistant` subcommand SHALL take a required `--deployment` flag ### Requirement: Import an assistant into a deployment -`langstar assistant import --deployment ` SHALL create an assistant on the deployment with the file's name, description, graph ID, configuration, context and metadata, and print the new assistant ID. It SHALL record the deployment and the new assistant ID in the destination record file, `.destinations.json` unless `--record ` names another. +`langstar assistant import --deployment ` SHALL choose a new assistant ID, write it as pending under the deployment in the destination record (`.destinations.json` unless `--record ` names another), then create the assistant with that `assistant_id`, `if_exists: do_nothing` and the file's name, description, graph ID, configuration, context and metadata, mark the entry created, and print the ID. A pending entry SHALL make the next import repeat the create. #### Scenario: First import - **WHEN** a user runs `langstar assistant import support-bot.json --deployment my-agent-prod` - **AND** the destination record has no entry for `my-agent-prod` - **THEN** langstar creates the assistant and prints its new ID -- **AND** `support-bot.destinations.json` holds the ID of the `my-agent-prod` deployment, with the new assistant ID under it +- **AND** `support-bot.destinations.json` holds the ID of the `my-agent-prod` deployment, with the new assistant ID under it, marked created + +#### Scenario: Import stopped after the create and before the record write + +- **WHEN** the destination record holds a pending assistant ID for `my-agent-prod` +- **AND** the deployment already has an assistant with that ID +- **AND** a user runs `langstar assistant import support-bot.json --deployment my-agent-prod` +- **THEN** langstar sends the create with that ID and `if_exists` set to `do_nothing`, so the deployment still holds one assistant from the file +- **AND** the destination record marks that ID created ### Requirement: Check the destination before importing an assistant @@ -88,7 +96,7 @@ Before it writes, `langstar assistant import` SHALL run the graph check and the ### Requirement: Find the earlier copy instead of creating another -When the destination record holds an assistant ID for the deployment, `langstar assistant import` SHALL compare the file with that assistant and create nothing. It SHALL exit with status 0 when they match, and list the differences and exit with status 3 when they differ, unless the user passes `--update`. When the recorded assistant no longer exists, it SHALL exit with status 1 and say so. +When the destination record holds a created assistant ID for the deployment, `langstar assistant import` SHALL compare the file with that assistant and create nothing. It SHALL exit with status 0 when they match, and list the differences and exit with status 3 when they differ, unless the user passes `--update`. When the recorded assistant no longer exists, it SHALL exit with status 1 and say so. #### Scenario: Import the same file again diff --git a/openspec/changes/export-import-prompts-and-assistants/tasks.md b/openspec/changes/export-import-prompts-and-assistants/tasks.md index 2dc53b895..739f298af 100644 --- a/openspec/changes/export-import-prompts-and-assistants/tasks.md +++ b/openspec/changes/export-import-prompts-and-assistants/tasks.md @@ -20,7 +20,7 @@ Each task in this group adds a public field to one of these SDK structs: `Prompt - [ ] 3.1 Add `description`, `model_config` and `model_provider` to `CommitManifestResponse`, and a commit read that sends `include=model` and returns that struct, to `sdk/src/prompts.rs`; verify with an httpmock test that asserts the query string and the parsed fields, and an integration test that reads a prompt with a model - [ ] 3.2 Add the commit's `id` to `CommitData` in `sdk/src/prompts.rs`, a call that lists a prompt's commits with their `id` and hash (`GET /api/v1/commits/{owner}/{repo}`), calls to list, create and move commit tags, which name their commit by that `id` in `POST /api/v1/repos/{owner}/{repo}/tags` and `PATCH /api/v1/repos/{owner}/{repo}/tags/{tag_name}`, and a call that deletes a tag with `DELETE /api/v1/repos/{owner}/{repo}/tags/{tag_name}`, which task 10.3 uses; verify with httpmock tests that assert `commit_id` in each create and move request body and the path of the delete request, and an integration test that pushes two commits, tags the first, finds the second commit's `id` through the commit list, moves the tag to it, reads the tag back through the SDK, deletes it, and asserts that the tag list no longer holds it -- [ ] 3.3 Add `context` and `version` to `Assistant`, `context` and `graph_id` to `UpdateAssistantRequest`, `context` to `CreateAssistantRequest`, and a `graph_id` filter to `AssistantSearchRequest`, in `sdk/src/assistants.rs`; verify with httpmock tests of the request bodies and an integration test that creates an assistant with context on the destination deployment from 2.4, reads it back with its version, finds it by graph ID, moves it from `test_graph` to `test_graph_alt`, and reads `test_graph_alt` back. The test runs on the destination deployment because the source deployment serves only `test_graph`, and task 2.4, which adds the destination fixture, merges before this group. +- [ ] 3.3 Add `context` and `version` to `Assistant`, `context` and `graph_id` to `UpdateAssistantRequest`, `context`, `assistant_id` and `if_exists` to `CreateAssistantRequest`, and a `graph_id` filter to `AssistantSearchRequest`, in `sdk/src/assistants.rs`; verify with httpmock tests of the request bodies and an integration test that creates an assistant with context on the destination deployment from 2.4, reads it back with its version, finds it by graph ID, moves it from `test_graph` to `test_graph_alt`, and reads `test_graph_alt` back. The test runs on the destination deployment because the source deployment serves only `test_graph`, and task 2.4, which adds the destination fixture, merges before this group. - [ ] 3.4 Add `description` to `CommitRequest`, `readme` and the repo's `tags` to `Prompt`, which `PromptClient::get` returns and export reads the repo details from, and a repo update that sends `PATCH /api/v1/repos/{owner}/{repo}` with `description`, `readme`, `tags` and `is_public`, to `sdk/src/prompts.rs`; verify with httpmock tests of the request bodies and of a `PromptClient::get` response that holds `readme` and `tags`, and an integration test that pushes a commit with a description and reads it back through the commit read from 3.1, then updates the repo's description, readme, tags and visibility and reads each back through `PromptClient::get` ## 4. Shared code for the prompt and assistant commands: the export file, the comparison and the report @@ -65,7 +65,7 @@ Each task in this group adds a public field to one of these SDK structs: `Prompt ## 10. assistant import -- [ ] 10.1 Add `langstar assistant import` with `--workspace-id`, the graph check and prompt check of `assistant verify` from 9.2 run before it writes, creation, read-back, the destination record and `--record`; verify with an integration test that creates a tagged prompt in the test workspace, imports an assistant that refers to it into the destination deployment and checks name, graph ID, configuration, context, metadata and prompt references through the Agent Server API, that `--record ` writes the destination record to that path, and that a file whose graph ID the deployment lacks and a file with a missing prompt tag each create no assistant and exit with status 3, and an httpmock test in which the assistant read back differs from the file, asserting that import exits with status 3, that the destination record holds the new assistant ID, and that a second import creates no assistant, and a CLI test that `assistant import` of a prompt export exits with status 1, names `kind` and the value `prompt`, and creates no assistant +- [ ] 10.1 Add `langstar assistant import` with `--workspace-id`, the graph check and prompt check of `assistant verify` from 9.2 run before it writes, creation, read-back, the destination record and `--record`; verify with an integration test that creates a tagged prompt in the test workspace, imports an assistant that refers to it into the destination deployment and checks name, graph ID, configuration, context, metadata and prompt references through the Agent Server API, that `--record ` writes the destination record to that path, and that a file whose graph ID the deployment lacks and a file with a missing prompt tag each create no assistant and exit with status 3, and an httpmock test in which the assistant read back differs from the file, asserting that import exits with status 3, that the destination record holds the new assistant ID, and that a second import creates no assistant, and a CLI test that `assistant import` of a prompt export exits with status 1, names `kind` and the value `prompt`, and creates no assistant, and an integration test that creates an assistant with a chosen ID on the destination deployment, writes a destination record that holds that ID as pending, imports the file, and asserts that the deployment holds one assistant from the file, with that ID, and that the record marks the ID created - [ ] 10.2 Add `--update`, `--assistant-id` and `--dry-run`; verify with integration tests that a repeated import creates no assistant, that `--update` changes the recorded one, moving it from `test_graph` to `test_graph_alt` among the fields, that `--update --assistant-id ` naming an assistant other than the recorded one updates that assistant, leaves the recorded one unchanged and writes the new ID to the destination record, that `--update` of an assistant with metadata `{"kept": 1, "stale": 2}` from a file with `{"kept": 1}` leaves `{"kept": 1}` or reports `/metadata/stale` and exits with status 3, whichever task 1.2 found the server does, that `--update` of an assistant with a description and context from a file with neither clears both or reports `/description` and `/context` and exits with status 3, whichever task 1.2 found the server does, that a deleted recorded assistant exits with status 1, and that `--dry-run` creates no assistant, changes none and leaves the destination record as it was - [ ] 10.3 Add an end-to-end test that imports prompts into the destination workspace, then imports an assistant that refers to them onto the destination deployment and runs `assistant verify`, passing `--workspace-id` set to the destination workspace to both assistant commands; verify that it passes, and that verify fails after deleting one prompt tag in the destination workspace through the tag delete call from 3.2 - [ ] 10.4 Document assistant import, the deployment scope of assistants, and how export and import relate to the `create`, `get` and `update` commands of [#728 Implement CRUD command structure (create, get, update) for prompt commands](https://github.com/codekiln/langstar/issues/728); verify that each documented command runs as written From 47e29a9732b044916b46b62ab699c2ff9c962a18 Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Tue, 13 Oct 2026 23:32:47 -0400 Subject: [PATCH 38/90] =?UTF-8?q?=F0=9F=93=9A=20docs(openspec):=20sort=20a?= =?UTF-8?q?=20commit's=20tags=20in=20the=20export=20file,=20give=20the=20s?= =?UTF-8?q?tructured=20output=20check=20its=20prompts,=20and=20name=20whic?= =?UTF-8?q?h=20record=20write=20import=20can=20lose?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit #807 Export, import, compare, and verify prompts and assistants across LangSmith environments Co-Authored-By: Claude Opus 5.5 --- .../changes/export-import-prompts-and-assistants/design.md | 4 ++-- .../specs/assistant/spec.md | 2 +- .../export-import-prompts-and-assistants/specs/prompt/spec.md | 2 +- .../changes/export-import-prompts-and-assistants/tasks.md | 3 ++- 4 files changed, 6 insertions(+), 5 deletions(-) diff --git a/openspec/changes/export-import-prompts-and-assistants/design.md b/openspec/changes/export-import-prompts-and-assistants/design.md index d8f201303..8b721e0b7 100644 --- a/openspec/changes/export-import-prompts-and-assistants/design.md +++ b/openspec/changes/export-import-prompts-and-assistants/design.md @@ -49,7 +49,7 @@ One prompt export looks like this: An assistant export has `kind: "assistant"`, a `source` with the deployment, assistant ID and version, an `assistant` object with `name`, `description`, `graph_id`, `config`, `context` and `metadata`, and `requires.prompts`. -The file holds no export time and langstar sorts its keys, so exporting an unchanged resource leaves Git with nothing to commit. +The file holds no export time, and langstar sorts its keys and the commit's tags, so exporting an unchanged resource leaves Git with nothing to commit. Without `--output`, export prints the file to stdout and nothing else, so `langstar prompt export my-prompt > my-prompt.json` writes a file import can read. Export writes its progress messages to stderr in every output format. [`OutputFormatter::print_message` in cli/src/output.rs:173](https://github.com/codekiln/langstar/blob/e4ae746d51f1f3f9ccf3fc97787148c3b4df2bce/cli/src/output.rs#L173), which `info` calls, prints to stdout in table mode, so export does not use it for those messages. @@ -61,7 +61,7 @@ Export reads the commit with `include=model`, the form the Go SDK uses (`referen ### Import records what it created in a separate file -Import writes the destination's IDs to `.destinations.json`, keyed by destination workspace ID for prompts and by deployment ID for assistants. The export file stays the same after any number of imports, and a fresh export never wipes out the record of earlier imports. A repeated import reads the record to find the copy it made. For a prompt, langstar finds the copy by its handle, and compares the commit hash in the record with the destination's latest commit to tell whether someone changed the prompt since the last import. For an assistant, whose ID differs on every deployment, the record is the only link from the file to its copy. Import writes the new commit hash or assistant ID to the record before it reads the copy back, so an import whose read-back finds a difference still records what it wrote. The next prompt import then compares the destination's latest commit with the commit langstar wrote, and reports a change only when someone else pushed one; the next assistant import finds that assistant instead of creating a second one. Import also chooses a new assistant's ID itself and writes it to the record, marked pending, before it sends the create request with that `assistant_id` and `if_exists` set to `do_nothing`, which the Agent Server's `AssistantCreate` accepts. If langstar stops between the create and the record write, the next import finds the pending entry and sends the same create again, which returns the assistant the first import made instead of creating a second one. +Import writes the destination's IDs to `.destinations.json`, keyed by destination workspace ID for prompts and by deployment ID for assistants. The export file stays the same after any number of imports, and a fresh export never wipes out the record of earlier imports. A repeated import reads the record to find the copy it made. For a prompt, langstar finds the copy by its handle, and compares the commit hash in the record with the destination's latest commit to tell whether someone changed the prompt since the last import. For an assistant, whose ID differs on every deployment, the record is the only link from the file to its copy. Import writes the new commit hash or assistant ID to the record before it reads the copy back, so an import whose read-back finds a difference still records what it wrote. The next prompt import then compares the destination's latest commit with the commit langstar wrote, and reports a change only when someone else pushed one; the next assistant import finds that assistant instead of creating a second one. Import also chooses a new assistant's ID itself and writes it to the record, marked pending, before it sends the create request with that `assistant_id` and `if_exists` set to `do_nothing`, which the Agent Server's `AssistantCreate` accepts. If langstar stops after the create and before it marks the entry created, the next import finds the pending entry and sends the same create again, which returns the assistant the first import made instead of creating a second one. ### Import changes an existing resource only with --update diff --git a/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md b/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md index 78c6ab744..228dd127e 100644 --- a/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md +++ b/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md @@ -75,7 +75,7 @@ Every `langstar assistant` subcommand SHALL take a required `--deployment` flag - **THEN** langstar creates the assistant and prints its new ID - **AND** `support-bot.destinations.json` holds the ID of the `my-agent-prod` deployment, with the new assistant ID under it, marked created -#### Scenario: Import stopped after the create and before the record write +#### Scenario: Import stopped after the create and before marking the ID created - **WHEN** the destination record holds a pending assistant ID for `my-agent-prod` - **AND** the deployment already has an assistant with that ID diff --git a/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md index 58ad3a034..3476aa287 100644 --- a/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md +++ b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md @@ -42,7 +42,7 @@ ### Requirement: Write the same file for the same commit -Exporting the same commit twice SHALL produce identical files when nobody changed the repo details or the tags that point at the commit in between, since those change without a new commit. The file SHALL hold no export time, and langstar SHALL write object keys in a fixed order. +Exporting the same commit twice SHALL produce identical files when nobody changed the repo details or the tags that point at the commit in between, since those change without a new commit. The file SHALL hold no export time, and langstar SHALL write object keys in a fixed order and the commit's tags sorted by name. #### Scenario: Export twice diff --git a/openspec/changes/export-import-prompts-and-assistants/tasks.md b/openspec/changes/export-import-prompts-and-assistants/tasks.md index 739f298af..133b3f181 100644 --- a/openspec/changes/export-import-prompts-and-assistants/tasks.md +++ b/openspec/changes/export-import-prompts-and-assistants/tasks.md @@ -37,6 +37,7 @@ Each task in this group adds a public field to one of these SDK structs: `Prompt - An integration test that exports, with `--commit prod`, a prompt whose `prod` tag points at an earlier commit, and asserts that `source.commit_hash` and the manifest are that earlier commit's. - An error test that exports a handle the workspace has no prompt for, and asserts that it exits with status 1, names the handle and writes no file. - A CLI test that runs `prompt export` without `--output`, parses all of stdout as one JSON document, and asserts that its `source.commit_hash` matches the SDK's read and that any progress message is on stderr. + - An integration test that gives one commit two tags, exports it twice, and asserts that the two files are identical. - [ ] 5.2 Document `prompt export` in `docs/usage/` and the README; verify that each documented command runs as written ## 6. prompt compare and verify @@ -49,7 +50,7 @@ Each task in this group adds a public field to one of these SDK structs: `Prompt - [ ] 7.1 Add `langstar prompt import`, which exits with status 1 and names `--workspace-id` when the flag is missing, runs the secret and saved model configuration checks of `prompt verify` from 6.2 before it writes, creates the repo, the commit and its tags, then reads the commit back; verify with integration tests that export from the test workspace, import into the destination workspace, and compare the full source and destination commits through the SDK with the model included, the commit description among them, and the repo's description, readme, tags and visibility, for every prompt shape from 5.1, the prompt without a model among them, that a file whose `requires.secrets` names a secret the destination workspace lacks creates no prompt, names the secret and exits with status 3, and a CLI test that runs import without `--workspace-id` while `LANGSMITH_WORKSPACE_ID` is set and asserts that it exits with status 1, names the missing flag and creates no prompt, and a CLI test that `prompt import` of an assistant export exits with status 1, names `kind` and the value `assistant`, and creates no prompt, and a CLI test that imports a file whose manifest refers to `ANTHROPIC_API_KEY` while `requires.secrets` holds only `OPENAI_API_KEY`, and another whose `requires.model_class` names a class other than the one at the end of the manifest's chat model `id`, asserting that each creates no prompt, names the field and exits with status 1 - [ ] 7.2 Add `--update`, `--dry-run`, `--record` and the destination record; verify with integration tests that a repeated import creates no commit, that a changed destination is refused without `--update`, that an import after someone pushed a commit to the destination since the recorded import says the destination changed since the last import and exits with status 3, that `--update` adds one commit with the right parent and the file's commit description, that `--update` against a destination whose repo description, readme, tags or visibility differ while the manifest matches updates the repo and adds no commit, that `--update` against a destination whose `prod` tag points at an earlier commit while the manifest matches moves the tag and adds no commit, that `--dry-run` lists the move, and that `--dry-run` writes nothing, and an httpmock test in which the commit read back lacks a field the file holds, asserting that import names that field's JSON Pointer path, exits with status 3, and leaves the new commit hash in the destination record -- [ ] 7.3 Add the structured output check as a mise task that pulls the imported prompt with `include_model=True` and a `secrets` map holding the model provider key read from the environment, invokes it and validates the result against the schema; verify by running it from the import integration test for both methods +- [ ] 7.3 Add the structured output check as a mise task that pulls the imported prompt with `include_model=True` and a `secrets` map holding the model provider key read from the environment, invokes it and validates the result against the schema; verify by having the import integration test create a `json_schema` prompt and a `function_calling` prompt, each saved with a small OpenAI model, import both into the destination workspace, and run the check on each - [ ] 7.4 Document prompt import, the workspace scope of prompts and setting up destination credentials; verify that each documented command runs as written ## 8. assistant export From a19f75b9ec6c2359d059d29c833e8561472b9217 Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Tue, 13 Oct 2026 23:47:27 -0400 Subject: [PATCH 39/90] =?UTF-8?q?=F0=9F=93=9A=20docs(openspec):=20resume?= =?UTF-8?q?=20a=20prompt=20import=20that=20stopped=20before=20its=20first?= =?UTF-8?q?=20commit,=20delete=20destination=20tags=20the=20file=20lacks?= =?UTF-8?q?=20on=20--update,=20and=20split=20the=20task=20test=20lists?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Split the destination record paragraph in design.md into three, list each task's tests as sub-bullets, complete the OpenAI key setup in design.md, and gate group 3's SDK integration tests on the integration-tests feature rather than a bare #[ignore]. #807 Export, import, compare, and verify prompts and assistants across LangSmith environments Co-Authored-By: Claude Opus 5.5 --- .../design.md | 19 ++++-- .../proposal.md | 4 +- .../specs/prompt/spec.md | 21 ++++++- .../tasks.md | 62 ++++++++++++++++--- 4 files changed, 89 insertions(+), 17 deletions(-) diff --git a/openspec/changes/export-import-prompts-and-assistants/design.md b/openspec/changes/export-import-prompts-and-assistants/design.md index 8b721e0b7..983c94b7d 100644 --- a/openspec/changes/export-import-prompts-and-assistants/design.md +++ b/openspec/changes/export-import-prompts-and-assistants/design.md @@ -61,11 +61,19 @@ Export reads the commit with `include=model`, the form the Go SDK uses (`referen ### Import records what it created in a separate file -Import writes the destination's IDs to `.destinations.json`, keyed by destination workspace ID for prompts and by deployment ID for assistants. The export file stays the same after any number of imports, and a fresh export never wipes out the record of earlier imports. A repeated import reads the record to find the copy it made. For a prompt, langstar finds the copy by its handle, and compares the commit hash in the record with the destination's latest commit to tell whether someone changed the prompt since the last import. For an assistant, whose ID differs on every deployment, the record is the only link from the file to its copy. Import writes the new commit hash or assistant ID to the record before it reads the copy back, so an import whose read-back finds a difference still records what it wrote. The next prompt import then compares the destination's latest commit with the commit langstar wrote, and reports a change only when someone else pushed one; the next assistant import finds that assistant instead of creating a second one. Import also chooses a new assistant's ID itself and writes it to the record, marked pending, before it sends the create request with that `assistant_id` and `if_exists` set to `do_nothing`, which the Agent Server's `AssistantCreate` accepts. If langstar stops after the create and before it marks the entry created, the next import finds the pending entry and sends the same create again, which returns the assistant the first import made instead of creating a second one. +Import writes the IDs of what it created to `.destinations.json`, keyed by destination workspace ID for prompts and by deployment ID for assistants. The export file stays the same after any number of imports, and a fresh export never wipes out the record of earlier imports. + +A repeated prompt import finds the copy by its handle and uses the record to tell whether someone changed it. Import writes the new commit hash to the record before it reads the commit back, so a read-back that finds a difference still leaves the hash in the record. The next import compares the destination's latest commit with that hash, and reports a change only when someone else pushed a commit after langstar's. + +An assistant's ID differs on every deployment, so the record is the only way a repeated import finds the assistant it made. Import chooses the new assistant's ID itself and writes it to the record, marked pending, before it sends the create request with that `assistant_id` and `if_exists` set to `do_nothing`, which the Agent Server's `AssistantCreate` accepts. After the create succeeds, import marks the entry created. If langstar stops before then, the next import finds the pending entry and sends the same create again, and the Agent Server returns the assistant the first import made. ### Import changes an existing resource only with --update -A prompt that exists in the destination gets a new commit only with `--update`, and its parent is the destination's latest commit, so LangSmith keeps the earlier history. Import sends the file's commit description with each commit it creates. With `--update`, import also sets the destination repo's description, readme, tags and visibility to the file's values through `PATCH /api/v1/repos/{owner}/{repo}` when they differ. When only those repo fields differ and the manifest matches, import updates the repo and adds no commit. A commit tag in the file that the destination lacks, or that points at another commit there, also counts as a difference; with `--update`, import creates or moves the tag to the destination commit that matches the file, and adds no commit when the latest commit already matches. The tag endpoints take a commit's `id`, which differs from its hash. The commit read and the repo read return no `id`, so import looks the commit up in the commit list, `GET /api/v1/commits/{owner}/{repo}`, whose entries carry both the `id` and the hash. +A prompt that exists in the destination gets a new commit only with `--update`, and its parent is the destination's latest commit, so LangSmith keeps the earlier history. Import sends the file's commit description with each commit it creates. With `--update`, import also sets the destination repo's description, readme, tags and visibility to the file's values through `PATCH /api/v1/repos/{owner}/{repo}` when they differ. When only those repo fields differ and the manifest matches, import updates the repo and adds no commit. + +Import treats the destination commit's tags as one more field to match. A tag in the file that the destination lacks, or that points at another commit there, is a difference, and so is a tag on the destination's matching commit that the file lacks. With `--update`, import creates or moves each of the file's tags to that commit and deletes each tag on it that the file lacks, through `DELETE /api/v1/repos/{owner}/{repo}/tags/{tag_name}`, so the read-back finds the same tags as the file. It adds no commit when the latest commit already matches. A tag on an older destination commit is not part of the comparison, so import leaves it alone. The tag endpoints take a commit's `id`, which differs from its hash. The commit read and the repo read return no `id`, so import looks the commit up in the commit list, `GET /api/v1/commits/{owner}/{repo}`, whose entries carry both the `id` and the hash. + +An import that stops after it creates the repo and before it pushes the commit leaves a prompt with no commit. The next import finds the prompt by its handle and, through the commit list, finds no commit. It then pushes the file's manifest as the first commit and creates the file's tags, as it does for a new prompt, without `--update`. The repo details it created hold the file's values already, and any that someone changed in between still need `--update`. With `--update`, import makes an existing assistant match the file and asks the Agent Server to remove what the file lacks. The Agent Server keeps the name, graph ID and description an update leaves out, and merges the metadata an update sends into the metadata the assistant already has, as `AssistantPatch` in `reference/openapi/langchain/agent-server/openapi.json` documents. So import sends the file's name, graph ID, configuration and context in full, an empty object for a configuration or context the file lacks, and a null value for each metadata key the destination has and the file lacks. `AssistantPatch` documents no way to clear a description, so in [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8) a developer sends a null and an empty description to the live server, along with a null metadata value and a replaced configuration and context, and records what the server keeps. Import then reads the assistant back, and any key or field the file lacks that is still there shows up as a difference, with exit status 3. The Agent Server saves each update as a new version of the assistant. @@ -73,7 +81,7 @@ Without `--update`, import lists the differences and exits with status 3. `--dry ### Compare reports differences by JSON path -Compare reads the prompt or assistant from the destination, puts it in the shape of the export file's `repo`, `commit` and `assistant` objects, and lists every value that differs from the file. Those objects hold no field the server sets, such as IDs, version numbers, timestamps, owner and like counts, so compare checks every key in them, including the `id` arrays inside a manifest that name the prompt and model classes, such as `ChatOpenAI`. The source environment's workspace, deployment and IDs sit in `source`, so they never show up as differences against a copy. Each difference names its JSON Pointer path and both values. For a prompt, the comparison covers the commit description, the commit tags and the commit each one points at, and the repo's description, readme, tags and visibility as well as the manifest. Import runs the same comparison after it writes, so a field LangSmith or the Agent Server drops shows up as a difference. +Compare reads the prompt or assistant from the destination, puts it in the shape of the export file's `repo`, `commit` and `assistant` objects, and lists every value that differs from the file. Those objects hold no field the server sets, such as IDs, version numbers, timestamps, owner and like counts, so compare checks every key in them, including the `id` arrays inside a manifest that name the prompt and model classes, such as `ChatOpenAI`. The source environment's workspace, deployment and IDs sit in `source`, so they never show up as differences against a copy. Each difference names its JSON Pointer path and both values. For a prompt, the comparison covers the commit description, the commit tags and the commit each one points at, any tag on the compared commit that the file lacks, and the repo's description, readme, tags and visibility as well as the manifest. Import runs the same comparison after it writes, so a field LangSmith or the Agent Server drops shows up as a difference. ### Verify checks what the copy needs to run @@ -99,7 +107,7 @@ Status 0 means the resource matches and every check passed. Status 3 means langs ### A Python script checks that an imported structured prompt still returns structured data -A mise task runs a short Python script that pulls the imported prompt with `include_model=True`, invokes it, and checks that the result matches the prompt's schema. The script passes `secrets={"OPENAI_API_KEY": }` to [`pull_prompt`](https://github.com/langchain-ai/langsmith-sdk/blob/47c553caf08c1989e2be3ff1886da3eb801f709f/python/langsmith/client.py#L10228), because the LangSmith Python SDK reads no secret from the environment unless asked and raises when a manifest holds a secret reference and no `secrets` map arrives. The script exists because the LangSmith API has no documented way to run a prompt with its model: the one prompt endpoint in the OpenAPI spec, `POST /api/v1/prompts/invoke_prompt`, takes message templates and inputs with no model or schema, and no LangSmith SDK calls it. In [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8), a developer calls that endpoint on the live API in case it does more than the spec says. The script needs a model provider API key, which codekiln has to create and add to fnox and CI before the test can run. +A mise task runs a short Python script that pulls the imported prompt with `include_model=True`, invokes it, and checks that the result matches the prompt's schema. The script passes `secrets={"OPENAI_API_KEY": }` to [`pull_prompt`](https://github.com/langchain-ai/langsmith-sdk/blob/47c553caf08c1989e2be3ff1886da3eb801f709f/python/langsmith/client.py#L10228), because the LangSmith Python SDK reads no secret from the environment unless asked and raises when a manifest holds a secret reference and no `secrets` map arrives. The script exists because the LangSmith API has no documented way to run a prompt with its model: the one prompt endpoint in the OpenAPI spec, `POST /api/v1/prompts/invoke_prompt`, takes message templates and inputs with no model or schema, and no LangSmith SDK calls it. In [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8), a developer calls that endpoint on the live API in case it does more than the spec says. The script needs an OpenAI API key. codekiln creates it and stores it in 1Password, in the repository's CI secrets and as the destination workspace's `OPENAI_API_KEY` secret, and in [export-import-prompts-and-assistants Task 2.1 Add the destination workspace and the model provider key](tasks.md#L12) a developer adds it to `fnox.toml` and the CI workflow. ### Export and import use the same SDK calls as the planned `prompt create`, `get` and `update` @@ -112,6 +120,7 @@ A mise task runs a short Python script that pulls the imported prompt with `incl - [langstar finds a deployment's Agent Server URL only in `source_config.custom_url` (`resolve_deployment_url` in `cli/src/deployment_utils.rs`), and a deployment built from GitHub has none, so every `langstar assistant` command, the new ones included, fails on such a deployment] → In [export-import-prompts-and-assistants Task 2.2 Return the URL of a deployment built from GitHub](tasks.md#L13), a developer adds the control plane's top-level `url` to the SDK's `Deployment`, which drops it today, and makes `resolve_deployment_url` use it, before anyone builds the assistant export, compare, verify and import commands. - [The SDK changes for export and import add public fields to `Deployment`, `Prompt`, `CommitManifestResponse`, `CommitData`, `CommitRequest`, `Assistant` and the assistant request structs, so a crate that builds one of those structs with a struct literal or destructures it with a pattern that names every field stops compiling] → Each pull request that adds a field updates every struct literal of that struct in this repository. [Should the SDK field additions count as a breaking change?](#9---should-the-sdk-field-additions-count-as-a-breaking-change) asks codekiln whether those pull requests also carry a `BREAKING CHANGE:` footer, which `docs/dev/git-scm-conventions.md` says makes the release a major version. - [A tag in the destination may already point at another commit] → `--update` moves the tag, and `--dry-run` lists the move. +- [`--update` deletes a tag on the matching destination commit that the file lacks, and a graph on a deployment may read that tag] → `--dry-run` lists each delete, and `assistant verify` reports a prompt tag an assistant refers to that the workspace no longer has. - [The control plane may refuse a second deployment from this repository with 409] → In [export-import-prompts-and-assistants Task 2.3 Register the test graph under a second graph ID](tasks.md#L14), a developer deploys the new `langgraph.json` once, before anyone builds the fixture. If the control plane answers with 409, the developer brings [Which second workspace and second deployment should the tests use?](#5---which-second-workspace-and-second-deployment-should-the-tests-use) back to codekiln. ## Resolved Questions @@ -137,7 +146,7 @@ None. ### 3 - How should the end-to-end test show that an imported structured prompt still returns structured data? > Context from the drafting agent, for question 3. -> The LangSmith OpenAPI spec's only prompt-invoke endpoint, `POST /api/v1/prompts/invoke_prompt`, takes message templates and inputs with no model or schema, and no SDK calls it. In [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8), a developer calls it on the live API. The draft runs a Python script through a mise task, using the LangSmith Python SDK and LangChain, which is how most users run a prompt. It needs a model provider key, such as `OPENAI_API_KEY`, which codekiln would create and add to fnox and CI. Building the model call into langstar as `prompt verify --invoke` would need langstar to call each provider's API itself. Recommendation: the Python script, with an OpenAI key and a small model. +> The LangSmith OpenAPI spec's only prompt-invoke endpoint, `POST /api/v1/prompts/invoke_prompt`, takes message templates and inputs with no model or schema, and no SDK calls it. In [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8), a developer calls it on the live API. The draft runs a Python script through a mise task, using the LangSmith Python SDK and LangChain, which is how most users run a prompt. It needs a model provider key, such as `OPENAI_API_KEY`, which codekiln would create and store in 1Password, the CI secrets and the destination workspace's secrets. Building the model call into langstar as `prompt verify --invoke` would need langstar to call each provider's API itself. Recommendation: the Python script, with an OpenAI key and a small model. diff --git a/openspec/changes/export-import-prompts-and-assistants/proposal.md b/openspec/changes/export-import-prompts-and-assistants/proposal.md index 3107e6fc1..ca6740b10 100644 --- a/openspec/changes/export-import-prompts-and-assistants/proposal.md +++ b/openspec/changes/export-import-prompts-and-assistants/proposal.md @@ -11,7 +11,7 @@ A prompt belongs to a workspace and an assistant belongs to a deployment, but bo ## What Changes - `langstar prompt export ` writes one JSON file holding a saved prompt commit as LangSmith returns it with the model included: message templates, input variables, output schema, structured output method, model binding and commit description, plus the repo's description, readme, tags and visibility, the commit tags, and where it came from. -- `langstar prompt import --workspace-id ` creates the prompt in the destination workspace, or adds a commit to an existing prompt, updates its repo details and moves its commit tags when the user passes `--update`. `--dry-run` shows what it would do. After writing, it reads the prompt back and reports any difference. +- `langstar prompt import --workspace-id ` creates the prompt in the destination workspace, or adds a commit to an existing prompt, updates its repo details and creates, moves or deletes its commit tags when the user passes `--update`. An import that stopped after creating the repo finishes on the next run. `--dry-run` shows what it would do. After writing, it reads the prompt back and reports any difference. - `langstar prompt compare ` reports the differences between the file and a prompt in a workspace. `langstar prompt verify ` also checks that the workspace has the secrets the prompt's model reads, and notes when the workspace has no saved model configuration for the model's provider. - `langstar assistant export`, `import`, `compare` and `verify` do the same for assistants on a deployment. The file holds the name, description, graph ID, configuration, context, metadata and any prompt names the assistant refers to. `verify` checks that the destination deployment has the graph and that the destination workspace has the referenced prompts. - Import records each destination ID (the new commit hash, or the new assistant ID) in a separate file next to the export. When a user imports the same file again, langstar reads that file to find the assistant it created earlier instead of creating another one, and to tell whether someone pushed a commit to the prompt since the last import. @@ -30,7 +30,7 @@ A prompt belongs to a workspace and an assistant belongs to a deployment, but bo ## Impact -- SDK: `sdk/src/prompts.rs` gains a commit read that asks for the model and returns it on `CommitManifestResponse`, a commit description on commit create, the new commit's `id`, the repo's readme and tags on `Prompt`, a repo update, a commit list that returns each commit's `id`, calls to list, create and move commit tags, and a tag delete that the end-to-end test uses. `sdk/src/assistants.rs` gains the `context` and `version` fields, `assistant_id` and `if_exists` on create, `graph_id` on update, and a search by graph ID. `sdk/src/deployments.rs` gains the deployment's `url`. Each new public field breaks a crate outside this repository that builds that SDK struct with a struct literal or destructures it with a pattern that names every field; codekiln decides in [Should the SDK field additions count as a breaking change?](design.md#9---should-the-sdk-field-additions-count-as-a-breaking-change) whether that makes the release a major version. +- SDK: `sdk/src/prompts.rs` gains a commit read that asks for the model and returns it on `CommitManifestResponse`, a commit description on commit create, the new commit's `id`, the repo's readme and tags on `Prompt`, a repo update, a commit list that returns each commit's `id`, calls to list, create and move commit tags, and a tag delete that `prompt import --update` and the end-to-end test use. `sdk/src/assistants.rs` gains the `context` and `version` fields, `assistant_id` and `if_exists` on create, `graph_id` on update, and a search by graph ID. `sdk/src/deployments.rs` gains the deployment's `url`. Each new public field breaks a crate outside this repository that builds that SDK struct with a struct literal or destructures it with a pattern that names every field; codekiln decides in [Should the SDK field additions count as a breaking change?](design.md#9---should-the-sdk-field-additions-count-as-a-breaking-change) whether that makes the release a major version. - CLI: new subcommands in `cli/src/commands/prompt.rs` and `cli/src/commands/assistant.rs`, and a shared module for the export file, the destination record, comparison and the exit status. `resolve_deployment_url` in `cli/src/deployment_utils.rs` also returns the Agent Server URL of a deployment built from GitHub, which every `langstar assistant` command needs. - Tests need a second workspace, a second deployment and one model provider key for the structured output check. codekiln creates the workspace and the key, and langstar's tests read them through `fnox.toml` and the CI workflow; the destination deployment fixture from [export-import-prompts-and-assistants Task 2.4 Add a destination deployment fixture](tasks.md#L15) creates the deployment whenever the cleanup workflow has deleted it, as the source fixture does today. Nobody knows yet whether the control plane accepts a second deployment from this repository. In [export-import-prompts-and-assistants Task 2.3 Register the test graph under a second graph ID](tasks.md#L14), a developer deploys the new `langgraph.json` once to find out, before anyone builds the fixture. - Docs: a new usage page on moving prompts and assistants between environments. diff --git a/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md index 3476aa287..0816e4acd 100644 --- a/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md +++ b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md @@ -109,6 +109,17 @@ When the destination workspace has no prompt with the file's handle, `langstar p - **THEN** langstar creates `my-prompt` in the destination with the file's manifest, model and tags - **AND** prints the new commit hash +### Requirement: Finish an import that stopped before its first commit + +When the destination workspace has a prompt with the file's handle and no commit, `langstar prompt import` SHALL push the file's manifest and commit description as the first commit and create the file's commit tags, as for a prompt that does not exist yet, without `--update`. Repo details that differ from the file SHALL still need `--update`. + +#### Scenario: Earlier import created the repo and stopped + +- **WHEN** an earlier import created `my-prompt` in the destination and stopped before it pushed the commit +- **AND** the user runs `langstar prompt import my-prompt.json --workspace-id ` again +- **THEN** langstar pushes the file's manifest as the prompt's first commit and creates the file's tags +- **AND** exits with status 0 + ### Requirement: Change an existing prompt only with --update When the destination prompt exists and its latest commit differs from the file, `langstar prompt import` SHALL change nothing and exit with status 3, listing the differences, unless the user passes `--update`. With `--update` it SHALL push the file's manifest and commit description as a new commit whose parent is the destination's latest commit, and move the file's tags to it. @@ -144,7 +155,7 @@ When the destination repo's description, readme, tags or visibility differ from ### Requirement: Move commit tags only with --update -When a commit tag in the file is missing in the destination or points at a different commit, `langstar prompt import` SHALL list the tag as a difference and exit with status 3. With `--update` it SHALL create or move the tag to the destination commit that matches the file, and add no commit when the latest commit already matches. +`prompt compare` and `prompt import` SHALL list as a difference a file tag the destination lacks or points at another commit, and a tag on the destination's matching commit that the file lacks; import SHALL then exit with status 3. With `--update`, import SHALL create or move each file tag to that commit, delete each tag on it the file lacks, and add no commit when the latest commit already matches. #### Scenario: Destination tag points at an older commit @@ -154,6 +165,14 @@ When a commit tag in the file is missing in the destination or points at a diffe - **THEN** langstar moves `prod` to the latest commit and adds no commit - **AND** exits with status 0 +#### Scenario: Destination commit has a tag the file lacks + +- **WHEN** the destination prompt's latest commit matches the file +- **AND** that commit carries a `staging` tag that the file's `commit.tags` lacks +- **AND** the user runs `langstar prompt import my-prompt.json --workspace-id --update` +- **THEN** langstar deletes the destination's `staging` tag and adds no commit +- **AND** exits with status 0 + ### Requirement: Record the destination commit After import writes a commit, langstar SHALL record the destination workspace, handle and commit hash in a destination record file, `.destinations.json` next to the export unless `--record ` names another, before it reads the commit back. Each import SHALL add or replace only the entry for its own destination workspace. diff --git a/openspec/changes/export-import-prompts-and-assistants/tasks.md b/openspec/changes/export-import-prompts-and-assistants/tasks.md index 133b3f181..65d76227a 100644 --- a/openspec/changes/export-import-prompts-and-assistants/tasks.md +++ b/openspec/changes/export-import-prompts-and-assistants/tasks.md @@ -16,10 +16,10 @@ Each numbered group below is one pull request, merged in the order listed, excep ## 3. SDK reads and writes the fields export and import need -Each task in this group adds a public field to one of these SDK structs: `Prompt` (the struct `PromptClient::get` returns, built with a struct literal in `test_prompt_serialization` in `sdk/src/prompts.rs`), `CommitManifestResponse` (the struct `PromptClient::get_commit` returns), `CommitData`, `CommitRequest`, `Assistant`, `CreateAssistantRequest`, `UpdateAssistantRequest` and `AssistantSearchRequest`. A crate that builds one of them with a struct literal, or destructures one with a pattern that names every field, stops compiling; nothing in this repository destructures them that way, so each task updates every struct literal of its struct in this repository. Whether this group's pull request and task 2.2's, which adds `url` to `Deployment`, carry a `BREAKING CHANGE:` footer waits on [Should the SDK field additions count as a breaking change?](design.md#9---should-the-sdk-field-additions-count-as-a-breaking-change). +Each task in this group adds a public field to one of these SDK structs: `Prompt` (the struct `PromptClient::get` returns, built with a struct literal in `test_prompt_serialization` in `sdk/src/prompts.rs`), `CommitManifestResponse` (the struct `PromptClient::get_commit` returns), `CommitData`, `CommitRequest`, `Assistant`, `CreateAssistantRequest`, `UpdateAssistantRequest` and `AssistantSearchRequest`. A crate that builds one of them with a struct literal, or destructures one with a pattern that names every field, stops compiling; nothing in this repository destructures them that way, so each task updates every struct literal of its struct in this repository. Whether this group's pull request and task 2.2's, which adds `url` to `Deployment`, carry a `BREAKING CHANGE:` footer waits on [Should the SDK field additions count as a breaking change?](design.md#9---should-the-sdk-field-additions-count-as-a-breaking-change). Each integration test in this group carries `#[cfg_attr(not(feature = "integration-tests"), ignore)]`, as the tests in `sdk/tests/structured_prompts_integration_test.rs` do, and not a bare `#[ignore]`. The SDK Integration Tests job in `.github/workflows/ci.yml` and `mise run check` both turn that feature on, so both run the test, while a bare `#[ignore]` would skip it in both. - [ ] 3.1 Add `description`, `model_config` and `model_provider` to `CommitManifestResponse`, and a commit read that sends `include=model` and returns that struct, to `sdk/src/prompts.rs`; verify with an httpmock test that asserts the query string and the parsed fields, and an integration test that reads a prompt with a model -- [ ] 3.2 Add the commit's `id` to `CommitData` in `sdk/src/prompts.rs`, a call that lists a prompt's commits with their `id` and hash (`GET /api/v1/commits/{owner}/{repo}`), calls to list, create and move commit tags, which name their commit by that `id` in `POST /api/v1/repos/{owner}/{repo}/tags` and `PATCH /api/v1/repos/{owner}/{repo}/tags/{tag_name}`, and a call that deletes a tag with `DELETE /api/v1/repos/{owner}/{repo}/tags/{tag_name}`, which task 10.3 uses; verify with httpmock tests that assert `commit_id` in each create and move request body and the path of the delete request, and an integration test that pushes two commits, tags the first, finds the second commit's `id` through the commit list, moves the tag to it, reads the tag back through the SDK, deletes it, and asserts that the tag list no longer holds it +- [ ] 3.2 Add the commit's `id` to `CommitData` in `sdk/src/prompts.rs`, a call that lists a prompt's commits with their `id` and hash (`GET /api/v1/commits/{owner}/{repo}`), calls to list, create and move commit tags, which name their commit by that `id` in `POST /api/v1/repos/{owner}/{repo}/tags` and `PATCH /api/v1/repos/{owner}/{repo}/tags/{tag_name}`, and a call that deletes a tag with `DELETE /api/v1/repos/{owner}/{repo}/tags/{tag_name}`, which `prompt import --update` from 7.2 and the end-to-end test in 10.3 use; verify with httpmock tests that assert `commit_id` in each create and move request body and the path of the delete request, and an integration test that pushes two commits, tags the first, finds the second commit's `id` through the commit list, moves the tag to it, reads the tag back through the SDK, deletes it, and asserts that the tag list no longer holds it - [ ] 3.3 Add `context` and `version` to `Assistant`, `context` and `graph_id` to `UpdateAssistantRequest`, `context`, `assistant_id` and `if_exists` to `CreateAssistantRequest`, and a `graph_id` filter to `AssistantSearchRequest`, in `sdk/src/assistants.rs`; verify with httpmock tests of the request bodies and an integration test that creates an assistant with context on the destination deployment from 2.4, reads it back with its version, finds it by graph ID, moves it from `test_graph` to `test_graph_alt`, and reads `test_graph_alt` back. The test runs on the destination deployment because the source deployment serves only `test_graph`, and task 2.4, which adds the destination fixture, merges before this group. - [ ] 3.4 Add `description` to `CommitRequest`, `readme` and the repo's `tags` to `Prompt`, which `PromptClient::get` returns and export reads the repo details from, and a repo update that sends `PATCH /api/v1/repos/{owner}/{repo}` with `description`, `readme`, `tags` and `is_public`, to `sdk/src/prompts.rs`; verify with httpmock tests of the request bodies and of a `PromptClient::get` response that holds `readme` and `tags`, and an integration test that pushes a commit with a description and reads it back through the commit read from 3.1, then updates the repo's description, readme, tags and visibility and reads each back through `PromptClient::get` @@ -42,32 +42,76 @@ Each task in this group adds a public field to one of these SDK structs: `Prompt ## 6. prompt compare and verify -- [ ] 6.1 Add `langstar prompt compare`; verify with integration tests that export a prompt, change its model temperature through the SDK, and assert the reported path, both values and exit status 3, and that a changed commit description and a changed repo readme are each reported the same way, and CLI tests that `langstar -f json prompt compare` prints one JSON object with `status`, `differences` and `checks` to stdout and its progress to stderr, and that `prompt compare` with a mistyped flag exits with status 2, and that `prompt compare` of a file holding `"format_version": 2` exits with status 1 and names `format_version` and the value 2, and that `prompt compare` of a file whose handle the workspace lacks exits with status 1 and names the handle +- [ ] 6.1 Add `langstar prompt compare`. Verify with these tests: + - An integration test that exports a prompt, changes its model temperature through the SDK, and asserts the reported path, both values and exit status 3. + - Integration tests that a changed commit description and a changed repo readme are each reported the same way. + - An integration test that exports a prompt, adds a tag to the exported commit through the SDK, and asserts that compare reports `/commit/tags` with both values and exits with status 3. + - A CLI test that `langstar -f json prompt compare` prints one JSON object with `status`, `differences` and `checks` to stdout and its progress to stderr. + - A CLI test that `prompt compare` with a mistyped flag exits with status 2. + - A CLI test that `prompt compare` of a file holding `"format_version": 2` exits with status 1 and names `format_version` and the value 2. + - A CLI test that `prompt compare` of a file whose handle the workspace lacks exits with status 1 and names the handle. - [ ] 6.2 Add `langstar prompt verify` with the secret check and the saved model configuration note, matching `requires.model_class` against the class name at the end of each configuration's settings `id`; verify with integration tests against a workspace that lacks a listed secret and one that has it, a test that a missing saved model configuration is reported and leaves the exit status at 0, a test that verify of a prompt without a model, whose `requires.model_class` is null, reports no saved model configuration note and exits with status 0, and a test that a file whose manifest refers to a secret its `requires.secrets` leaves out exits with status 1 and names `requires.secrets` - [ ] 6.3 Document compare and verify; verify that each documented command runs as written ## 7. prompt import -- [ ] 7.1 Add `langstar prompt import`, which exits with status 1 and names `--workspace-id` when the flag is missing, runs the secret and saved model configuration checks of `prompt verify` from 6.2 before it writes, creates the repo, the commit and its tags, then reads the commit back; verify with integration tests that export from the test workspace, import into the destination workspace, and compare the full source and destination commits through the SDK with the model included, the commit description among them, and the repo's description, readme, tags and visibility, for every prompt shape from 5.1, the prompt without a model among them, that a file whose `requires.secrets` names a secret the destination workspace lacks creates no prompt, names the secret and exits with status 3, and a CLI test that runs import without `--workspace-id` while `LANGSMITH_WORKSPACE_ID` is set and asserts that it exits with status 1, names the missing flag and creates no prompt, and a CLI test that `prompt import` of an assistant export exits with status 1, names `kind` and the value `assistant`, and creates no prompt, and a CLI test that imports a file whose manifest refers to `ANTHROPIC_API_KEY` while `requires.secrets` holds only `OPENAI_API_KEY`, and another whose `requires.model_class` names a class other than the one at the end of the manifest's chat model `id`, asserting that each creates no prompt, names the field and exits with status 1 -- [ ] 7.2 Add `--update`, `--dry-run`, `--record` and the destination record; verify with integration tests that a repeated import creates no commit, that a changed destination is refused without `--update`, that an import after someone pushed a commit to the destination since the recorded import says the destination changed since the last import and exits with status 3, that `--update` adds one commit with the right parent and the file's commit description, that `--update` against a destination whose repo description, readme, tags or visibility differ while the manifest matches updates the repo and adds no commit, that `--update` against a destination whose `prod` tag points at an earlier commit while the manifest matches moves the tag and adds no commit, that `--dry-run` lists the move, and that `--dry-run` writes nothing, and an httpmock test in which the commit read back lacks a field the file holds, asserting that import names that field's JSON Pointer path, exits with status 3, and leaves the new commit hash in the destination record +- [ ] 7.1 Add `langstar prompt import`, which exits with status 1 and names `--workspace-id` when the flag is missing, runs the secret and saved model configuration checks of `prompt verify` from 6.2 before it writes, creates the repo, the commit and its tags, then reads the commit back. When the destination has a prompt with the file's handle and no commit, import pushes the first commit and creates the tags, as for a new prompt. Verify with these tests: + - Integration tests that export each prompt shape from 5.1, the prompt without a model among them, from the test workspace, import it into the destination workspace, and compare the source and destination through the SDK with the model included: the manifest, the commit description, and the repo's description, readme, tags and visibility. + - An integration test that imports a file whose `requires.secrets` names a secret the destination workspace lacks, and asserts that it creates no prompt, names the secret and exits with status 3. + - An integration test that creates a repo with the file's handle and repo details and no commit in the destination workspace, as an import that stopped after creating the repo leaves it, then imports the file without `--update`, and asserts that the prompt holds one commit with the file's manifest and the file's tags, and that import exits with status 0. + - A CLI test that runs import without `--workspace-id` while `LANGSMITH_WORKSPACE_ID` is set, and asserts that it exits with status 1, names the missing flag and creates no prompt. + - A CLI test that runs `prompt import` on an assistant export, and asserts that it exits with status 1, names `kind` and the value `assistant`, and creates no prompt. + - CLI tests that import a file whose manifest refers to `ANTHROPIC_API_KEY` while `requires.secrets` holds only `OPENAI_API_KEY`, and a file whose `requires.model_class` names a class other than the one at the end of the manifest's chat model `id`, and assert that each creates no prompt, names the field and exits with status 1. +- [ ] 7.2 Add `--update`, `--dry-run`, `--record` and the destination record. Verify with these tests: + - An integration test that a repeated import creates no commit. + - An integration test that a changed destination is refused without `--update`. + - An integration test that an import after someone pushed a commit to the destination since the recorded import says the destination changed since the last import and exits with status 3. + - An integration test that `--update` adds one commit with the right parent and the file's commit description. + - An integration test that `--update` against a destination whose repo description, readme, tags or visibility differ while the manifest matches updates the repo and adds no commit. + - An integration test that `--update` against a destination whose `prod` tag points at an earlier commit while the manifest matches moves the tag and adds no commit, and that `--dry-run` lists the move. + - An integration test that `--update` against a destination whose latest commit matches the file and carries a `staging` tag the file lacks deletes `staging`, adds no commit and exits with status 0, and that `--dry-run` lists the delete. + - An integration test that `--dry-run` writes nothing. + - An httpmock test in which the commit read back lacks a field the file holds, asserting that import names that field's JSON Pointer path, exits with status 3, and leaves the new commit hash in the destination record. - [ ] 7.3 Add the structured output check as a mise task that pulls the imported prompt with `include_model=True` and a `secrets` map holding the model provider key read from the environment, invokes it and validates the result against the schema; verify by having the import integration test create a `json_schema` prompt and a `function_calling` prompt, each saved with a small OpenAI model, import both into the destination workspace, and run the check on each - [ ] 7.4 Document prompt import, the workspace scope of prompts and setting up destination credentials; verify that each documented command runs as written ## 8. assistant export -- [ ] 8.1 Add `langstar assistant export` with `--output` and `--prompt-ref`; verify with integration tests that create an assistant with configuration, context and metadata through the SDK and assert every field of the exported file, that an export with `--prompt-ref /config/configurable/system_prompt` of an assistant whose value there is `support-system:prod` writes that pointer and `support-system:prod` under `requires.prompts`, an error test for a pointer that names nothing, an error test that exporting an assistant ID the deployment lacks exits with status 1, names the ID and writes no file, and a CLI test that runs `assistant export` without `--output`, parses all of stdout as one JSON document, and asserts that its `source.assistant_id` is the exported assistant's ID and that any progress message is on stderr +- [ ] 8.1 Add `langstar assistant export` with `--output` and `--prompt-ref`. Verify with these tests: + - An integration test that creates an assistant with configuration, context and metadata through the SDK and asserts every field of the exported file. + - An integration test that exports, with `--prompt-ref /config/configurable/system_prompt`, an assistant whose value there is `support-system:prod`, and asserts that the file holds that pointer and `support-system:prod` under `requires.prompts`. + - An error test for a pointer that names nothing. + - An error test that exporting an assistant ID the deployment lacks exits with status 1, names the ID and writes no file. + - A CLI test that runs `assistant export` without `--output`, parses all of stdout as one JSON document, and asserts that its `source.assistant_id` is the exported assistant's ID and that any progress message is on stderr. - [ ] 8.2 Document assistant export; verify that each documented command runs as written ## 9. assistant compare and verify -- [ ] 9.1 Add `langstar assistant compare` with `--assistant-id`, and the lookup of the recorded assistant in the destination record from 4.1; verify with an integration test that exports an assistant, changes its metadata through the SDK, runs compare with `--assistant-id` set to that assistant, and asserts the reported path and exit status 3, and a unit test that compare reads the assistant ID from a destination record written by the 4.1 types, and an integration test that `assistant compare` with no destination record entry and no `--assistant-id` exits with status 1 and names the deployment; add the check, shared by assistant compare, verify and import, that each pointer under `requires.prompts` still resolves in the file's `assistant` object to the prompt name recorded with it, and verify with a unit test that a file whose configuration names another prompt at that pointer exits with status 1 and names the pointer and both values +- [ ] 9.1 Add `langstar assistant compare` with `--assistant-id`, the lookup of the recorded assistant in the destination record from 4.1, and the check, shared by assistant compare, verify and import, that each pointer under `requires.prompts` still resolves in the file's `assistant` object to the prompt name recorded with it. Verify with these tests: + - An integration test that exports an assistant, changes its metadata through the SDK, runs compare with `--assistant-id` set to that assistant, and asserts the reported path and exit status 3. + - A unit test that compare reads the assistant ID from a destination record written by the 4.1 types. + - An integration test that `assistant compare` with no destination record entry and no `--assistant-id` exits with status 1 and names the deployment. + - A unit test that a file whose configuration names another prompt at a pointer under `requires.prompts` exits with status 1 and names the pointer and both values. - [ ] 9.2 Add `langstar assistant verify` with `--assistant-id`, `--workspace-id` for the workspace that holds the prompts, the graph check, and the prompt check, which reads `GET /api/v1/commits/{owner}/{repo}/{tag}`; verify with integration tests that run verify with `--assistant-id` against an exported assistant, one with a file whose graph ID the deployment lacks, one with a missing prompt tag, and one that passes `--workspace-id` set to the destination workspace, and a CLI test that `langstar -f json assistant verify` prints one JSON object with `status`, `differences` and `checks` to stdout - [ ] 9.3 Document assistant compare and verify; verify that each documented command runs as written ## 10. assistant import -- [ ] 10.1 Add `langstar assistant import` with `--workspace-id`, the graph check and prompt check of `assistant verify` from 9.2 run before it writes, creation, read-back, the destination record and `--record`; verify with an integration test that creates a tagged prompt in the test workspace, imports an assistant that refers to it into the destination deployment and checks name, graph ID, configuration, context, metadata and prompt references through the Agent Server API, that `--record ` writes the destination record to that path, and that a file whose graph ID the deployment lacks and a file with a missing prompt tag each create no assistant and exit with status 3, and an httpmock test in which the assistant read back differs from the file, asserting that import exits with status 3, that the destination record holds the new assistant ID, and that a second import creates no assistant, and a CLI test that `assistant import` of a prompt export exits with status 1, names `kind` and the value `prompt`, and creates no assistant, and an integration test that creates an assistant with a chosen ID on the destination deployment, writes a destination record that holds that ID as pending, imports the file, and asserts that the deployment holds one assistant from the file, with that ID, and that the record marks the ID created -- [ ] 10.2 Add `--update`, `--assistant-id` and `--dry-run`; verify with integration tests that a repeated import creates no assistant, that `--update` changes the recorded one, moving it from `test_graph` to `test_graph_alt` among the fields, that `--update --assistant-id ` naming an assistant other than the recorded one updates that assistant, leaves the recorded one unchanged and writes the new ID to the destination record, that `--update` of an assistant with metadata `{"kept": 1, "stale": 2}` from a file with `{"kept": 1}` leaves `{"kept": 1}` or reports `/metadata/stale` and exits with status 3, whichever task 1.2 found the server does, that `--update` of an assistant with a description and context from a file with neither clears both or reports `/description` and `/context` and exits with status 3, whichever task 1.2 found the server does, that a deleted recorded assistant exits with status 1, and that `--dry-run` creates no assistant, changes none and leaves the destination record as it was +- [ ] 10.1 Add `langstar assistant import` with `--workspace-id`, the graph check and prompt check of `assistant verify` from 9.2 run before it writes, creation, read-back, the destination record and `--record`. Verify with these tests: + - An integration test that creates a tagged prompt in the test workspace, imports an assistant that refers to it into the destination deployment, and checks the name, graph ID, configuration, context, metadata and prompt references through the Agent Server API. + - An integration test that `--record ` writes the destination record to that path. + - Integration tests that a file whose graph ID the deployment lacks, and a file with a missing prompt tag, each create no assistant and exit with status 3. + - An httpmock test in which the assistant read back differs from the file, asserting that import exits with status 3, that the destination record holds the new assistant ID, and that a second import creates no assistant. + - A CLI test that `assistant import` of a prompt export exits with status 1, names `kind` and the value `prompt`, and creates no assistant. + - An integration test that creates an assistant with a chosen ID on the destination deployment, writes a destination record that holds that ID as pending, imports the file, and asserts that the deployment holds one assistant from the file, with that ID, and that the record marks the ID created. +- [ ] 10.2 Add `--update`, `--assistant-id` and `--dry-run`. Verify with these tests: + - An integration test that a repeated import creates no assistant. + - An integration test that `--update` changes the recorded assistant, moving it from `test_graph` to `test_graph_alt` among the fields. + - An integration test that `--update --assistant-id ` naming an assistant other than the recorded one updates that assistant, leaves the recorded one unchanged and writes the new ID to the destination record. + - An integration test that `--update` of an assistant with metadata `{"kept": 1, "stale": 2}` from a file with `{"kept": 1}` leaves `{"kept": 1}`, or reports `/metadata/stale` and exits with status 3, whichever task 1.2 found the server does. + - An integration test that `--update` of an assistant with a description and context from a file with neither clears both, or reports `/description` and `/context` and exits with status 3, whichever task 1.2 found the server does. + - An integration test that a deleted recorded assistant exits with status 1. + - An integration test that `--dry-run` creates no assistant, changes none and leaves the destination record as it was. - [ ] 10.3 Add an end-to-end test that imports prompts into the destination workspace, then imports an assistant that refers to them onto the destination deployment and runs `assistant verify`, passing `--workspace-id` set to the destination workspace to both assistant commands; verify that it passes, and that verify fails after deleting one prompt tag in the destination workspace through the tag delete call from 3.2 - [ ] 10.4 Document assistant import, the deployment scope of assistants, and how export and import relate to the `create`, `get` and `update` commands of [#728 Implement CRUD command structure (create, get, update) for prompt commands](https://github.com/codekiln/langstar/issues/728); verify that each documented command runs as written From 0da5ce91aca378caabecf5f9da189f2df0462396 Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Wed, 14 Oct 2026 00:00:58 -0400 Subject: [PATCH 40/90] =?UTF-8?q?=F0=9F=93=9A=20docs(openspec):=20leave=20?= =?UTF-8?q?model=5Fconfig=20and=20model=5Fprovider=20out=20of=20compare,?= =?UTF-8?q?=20cite=20what=20model=5Fprovider=20holds,=20and=20split=20task?= =?UTF-8?q?s=202.2,=204.2,=206.2=20and=209.2=20into=20lists?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit #807 Export, import, compare, and verify prompts and assistants across LangSmith environments Co-Authored-By: Claude Opus 5.5 --- .../design.md | 12 +++++---- .../proposal.md | 2 +- .../specs/prompt/spec.md | 9 +++++-- .../tasks.md | 27 +++++++++++++++---- 4 files changed, 37 insertions(+), 13 deletions(-) diff --git a/openspec/changes/export-import-prompts-and-assistants/design.md b/openspec/changes/export-import-prompts-and-assistants/design.md index 983c94b7d..1488c81de 100644 --- a/openspec/changes/export-import-prompts-and-assistants/design.md +++ b/openspec/changes/export-import-prompts-and-assistants/design.md @@ -57,7 +57,7 @@ Import, compare and verify read a file only when its `format_version` is 1 and i ### Export asks LangSmith for the model -Export reads the commit with `include=model`, the form the Go SDK uses (`references/langsmith-go.md`). The Python SDK sends the older `include_model=true`, and the LangSmith OpenAPI spec accepts both. With the model included, the manifest wraps the prompt and the chat model with its settings in one constructor, and the response also carries `model_config` and `model_provider` beside the manifest (`references/langsmith-sdk.md`). No field of the create-commit request holds the model, so the model reaches the destination inside the manifest. In [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8), a developer pushes that manifest to a second workspace and records whether LangSmith returns the same model, and which outer `id` the live API returns. +Export reads the commit with `include=model`, the form the Go SDK uses (`references/langsmith-go.md`). The Python SDK sends the older `include_model=true`, and the LangSmith OpenAPI spec accepts both. With the model included, the manifest wraps the prompt and the chat model with its settings in one constructor, and the response also carries `model_config` and `model_provider` beside the manifest (`references/langsmith-sdk.md`). No field of the create-commit request holds the model, so the model reaches the destination inside the manifest, and export keeps `model_config` and `model_provider` in the file only for the reader. In [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8), a developer pushes that manifest to a second workspace and records whether LangSmith returns the same manifest, `model_config` and `model_provider`, and which outer `id` the live API returns. ### Import records what it created in a separate file @@ -81,13 +81,15 @@ Without `--update`, import lists the differences and exits with status 3. `--dry ### Compare reports differences by JSON path -Compare reads the prompt or assistant from the destination, puts it in the shape of the export file's `repo`, `commit` and `assistant` objects, and lists every value that differs from the file. Those objects hold no field the server sets, such as IDs, version numbers, timestamps, owner and like counts, so compare checks every key in them, including the `id` arrays inside a manifest that name the prompt and model classes, such as `ChatOpenAI`. The source environment's workspace, deployment and IDs sit in `source`, so they never show up as differences against a copy. Each difference names its JSON Pointer path and both values. For a prompt, the comparison covers the commit description, the commit tags and the commit each one points at, any tag on the compared commit that the file lacks, and the repo's description, readme, tags and visibility as well as the manifest. Import runs the same comparison after it writes, so a field LangSmith or the Agent Server drops shows up as a difference. +Compare reads the prompt or assistant from the destination, puts it in the shape of the export file's `repo`, `commit` and `assistant` objects, and lists every value that differs from the file. Apart from `commit.model_config` and `commit.model_provider`, which the next paragraph covers, those objects hold no field the server sets, such as IDs, version numbers, timestamps, owner and like counts, so compare checks every other key in them, including the `id` arrays inside a manifest that name the prompt and model classes, such as `ChatOpenAI`. The source environment's workspace, deployment and IDs sit in `source`, so they never show up as differences against a copy. Each difference names its JSON Pointer path and both values. For a prompt, the comparison covers the commit description, the commit tags and the commit each one points at, any tag on the compared commit that the file lacks, and the repo's description, readme, tags and visibility as well as the manifest. Import runs the same comparison after it writes, so a field LangSmith or the Agent Server drops shows up as a difference. + +Compare and import's read-back leave out `commit.model_config` and `commit.model_provider`, because import cannot set them. No field of the create-commit request holds either one, so the destination fills them itself, and if it filled them differently from the source, every prompt import's read-back would report a difference and exit with status 3. The model reaches the destination inside the manifest, so compare checks the model there, through the chat model constructor's `id` array and settings. ### Verify checks what the copy needs to run Prompt verify checks that the workspace holds every secret the prompt's model reads. Export lists those names in `requires.secrets` by walking the manifest for LangChain secret references, `{"lc": 1, "type": "secret", "id": ["OPENAI_API_KEY"]}`, the shape the Python SDK's `_manifest_has_secrets` matches. Verify looks each name up among the workspace's secrets (`GET /api/v1/workspaces/current/secrets`, which `langstar secrets list` already calls) and prints the name of each missing secret and no secret value. -Prompt verify also notes when the workspace has no saved model configuration for the prompt's model, and leaves the exit status alone, because the prompt runs with the model its manifest carries. Export takes the model's class name, such as `ChatOpenAI`, from the last element of the chat model constructor's `id` in the manifest, such as `["langchain", "chat_models", "openai", "ChatOpenAI"]`, and stores it in `requires.model_class`. Verify looks for a saved model configuration (`GET /api/v1/playground-settings`, which `langstar model-config list` calls) whose settings `id` ends in that class name. LangSmith also returns a `model_provider` field beside the manifest, which the file keeps as `commit.model_provider`; no SDK or spec shows what it holds, so in [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8) a developer records the value the live API returns. [#807 Export, import, compare, and verify prompts and assistants across LangSmith environments](https://github.com/codekiln/langstar/issues/807) asks for the note, and [Should prompt verify report a missing saved model configuration?](#7---should-prompt-verify-report-a-missing-saved-model-configuration) asks whether to keep it. A prompt saved without a model has no chat model constructor in its manifest, so export writes `requires.model_class` as null, and verify and import skip the saved model configuration check for it and still run the secret check. +Prompt verify also notes when the workspace has no saved model configuration for the prompt's model, and leaves the exit status alone, because the prompt runs with the model its manifest carries. Export takes the model's class name, such as `ChatOpenAI`, from the last element of the chat model constructor's `id` in the manifest, such as `["langchain", "chat_models", "openai", "ChatOpenAI"]`, and stores it in `requires.model_class`. Verify looks for a saved model configuration (`GET /api/v1/playground-settings`, which `langstar model-config list` calls) whose settings `id` ends in that class name. LangSmith also returns a `model_provider` field beside the manifest, which the file keeps as `commit.model_provider`. The LangSmith Python SDK's [`PromptCommit`](https://github.com/langchain-ai/langsmith-sdk/blob/47c553caf08c1989e2be3ff1886da3eb801f709f/python/langsmith/schemas.py#L1054) calls it the model provider and gives `ChatOpenAI` as its example, which is the class name export writes to `requires.model_class`. In [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8), a developer records the value the live API returns and whether it matches `requires.model_class`. [#807 Export, import, compare, and verify prompts and assistants across LangSmith environments](https://github.com/codekiln/langstar/issues/807) asks for the note, and [Should prompt verify report a missing saved model configuration?](#7---should-prompt-verify-report-a-missing-saved-model-configuration) asks whether to keep it. A prompt saved without a model has no chat model constructor in its manifest, so export writes `requires.model_class` as null, and verify and import skip the saved model configuration check for it and still run the secret check. Import and verify work out `requires` again from the file's manifest: the secret names, and the model class or null. When either differs from what the file holds, they write nothing, name the field and exit with status 1. This catches a manifest someone edited in Git without updating `requires`. @@ -121,7 +123,7 @@ A mise task runs a short Python script that pulls the imported prompt with `incl - [The SDK changes for export and import add public fields to `Deployment`, `Prompt`, `CommitManifestResponse`, `CommitData`, `CommitRequest`, `Assistant` and the assistant request structs, so a crate that builds one of those structs with a struct literal or destructures it with a pattern that names every field stops compiling] → Each pull request that adds a field updates every struct literal of that struct in this repository. [Should the SDK field additions count as a breaking change?](#9---should-the-sdk-field-additions-count-as-a-breaking-change) asks codekiln whether those pull requests also carry a `BREAKING CHANGE:` footer, which `docs/dev/git-scm-conventions.md` says makes the release a major version. - [A tag in the destination may already point at another commit] → `--update` moves the tag, and `--dry-run` lists the move. - [`--update` deletes a tag on the matching destination commit that the file lacks, and a graph on a deployment may read that tag] → `--dry-run` lists each delete, and `assistant verify` reports a prompt tag an assistant refers to that the workspace no longer has. -- [The control plane may refuse a second deployment from this repository with 409] → In [export-import-prompts-and-assistants Task 2.3 Register the test graph under a second graph ID](tasks.md#L14), a developer deploys the new `langgraph.json` once, before anyone builds the fixture. If the control plane answers with 409, the developer brings [Which second workspace and second deployment should the tests use?](#5---which-second-workspace-and-second-deployment-should-the-tests-use) back to codekiln. +- [The control plane may refuse a second deployment from this repository with 409] → In [export-import-prompts-and-assistants Task 2.3 Register the test graph under a second graph ID](tasks.md#L18), a developer deploys the new `langgraph.json` once, before anyone builds the fixture. If the control plane answers with 409, the developer brings [Which second workspace and second deployment should the tests use?](#5---which-second-workspace-and-second-deployment-should-the-tests-use) back to codekiln. ## Resolved Questions @@ -160,7 +162,7 @@ None. ### 5 - Which second workspace and second deployment should the tests use? > Context from the drafting agent, for question 5. -> The import tests need a workspace and a deployment to import into. The draft has codekiln create a second workspace in the test organization, which the existing API key can reach, and the tests read its ID from `LANGSTAR_TEST_DEST_WORKSPACE_ID`. The destination deployment fixture from [export-import-prompts-and-assistants Task 2.4 Add a destination deployment fixture](tasks.md#L15) deploys the test graph a second time from a `langgraph.json` of its own, which registers the graph as both `test_graph` and `test_graph_alt` so the update tests can move an assistant from one graph to the other. Nobody knows yet whether the control plane accepts that deployment. It answers a second deployment of the same agent environment with 409 "A deployment already exists for this agent environment", and no document says whether a second `langgraph.json` in the same repository makes a new agent environment. In [export-import-prompts-and-assistants Task 2.3 Register the test graph under a second graph ID](tasks.md#L14), a developer deploys the new `langgraph.json` once to find out. If the control plane answers with the 409, the developer stops before task 2.4 and brings this question back to codekiln with what the control plane returned. `.github/workflows/cleanup-test-deployments.yml` deletes each test deployment once it is four hours old and the fixture rebuilds it on the next CI run, so the first CI run after each cleanup waits for two deployments to build instead of one. Recommendation: a second workspace in the same organization, which codekiln creates, and one destination deployment that the fixture from task 2.4 rebuilds whenever the cleanup workflow has deleted it. +> The import tests need a workspace and a deployment to import into. The draft has codekiln create a second workspace in the test organization, which the existing API key can reach, and the tests read its ID from `LANGSTAR_TEST_DEST_WORKSPACE_ID`. The destination deployment fixture from [export-import-prompts-and-assistants Task 2.4 Add a destination deployment fixture](tasks.md#L19) deploys the test graph a second time from a `langgraph.json` of its own, which registers the graph as both `test_graph` and `test_graph_alt` so the update tests can move an assistant from one graph to the other. Nobody knows yet whether the control plane accepts that deployment. It answers a second deployment of the same agent environment with 409 "A deployment already exists for this agent environment", and no document says whether a second `langgraph.json` in the same repository makes a new agent environment. In [export-import-prompts-and-assistants Task 2.3 Register the test graph under a second graph ID](tasks.md#L18), a developer deploys the new `langgraph.json` once to find out. If the control plane answers with the 409, the developer stops before task 2.4 and brings this question back to codekiln with what the control plane returned. `.github/workflows/cleanup-test-deployments.yml` deletes each test deployment once it is four hours old and the fixture rebuilds it on the next CI run, so the first CI run after each cleanup waits for two deployments to build instead of one. Recommendation: a second workspace in the same organization, which codekiln creates, and one destination deployment that the fixture from task 2.4 rebuilds whenever the cleanup workflow has deleted it. diff --git a/openspec/changes/export-import-prompts-and-assistants/proposal.md b/openspec/changes/export-import-prompts-and-assistants/proposal.md index ca6740b10..5b59b07c4 100644 --- a/openspec/changes/export-import-prompts-and-assistants/proposal.md +++ b/openspec/changes/export-import-prompts-and-assistants/proposal.md @@ -32,7 +32,7 @@ A prompt belongs to a workspace and an assistant belongs to a deployment, but bo - SDK: `sdk/src/prompts.rs` gains a commit read that asks for the model and returns it on `CommitManifestResponse`, a commit description on commit create, the new commit's `id`, the repo's readme and tags on `Prompt`, a repo update, a commit list that returns each commit's `id`, calls to list, create and move commit tags, and a tag delete that `prompt import --update` and the end-to-end test use. `sdk/src/assistants.rs` gains the `context` and `version` fields, `assistant_id` and `if_exists` on create, `graph_id` on update, and a search by graph ID. `sdk/src/deployments.rs` gains the deployment's `url`. Each new public field breaks a crate outside this repository that builds that SDK struct with a struct literal or destructures it with a pattern that names every field; codekiln decides in [Should the SDK field additions count as a breaking change?](design.md#9---should-the-sdk-field-additions-count-as-a-breaking-change) whether that makes the release a major version. - CLI: new subcommands in `cli/src/commands/prompt.rs` and `cli/src/commands/assistant.rs`, and a shared module for the export file, the destination record, comparison and the exit status. `resolve_deployment_url` in `cli/src/deployment_utils.rs` also returns the Agent Server URL of a deployment built from GitHub, which every `langstar assistant` command needs. -- Tests need a second workspace, a second deployment and one model provider key for the structured output check. codekiln creates the workspace and the key, and langstar's tests read them through `fnox.toml` and the CI workflow; the destination deployment fixture from [export-import-prompts-and-assistants Task 2.4 Add a destination deployment fixture](tasks.md#L15) creates the deployment whenever the cleanup workflow has deleted it, as the source fixture does today. Nobody knows yet whether the control plane accepts a second deployment from this repository. In [export-import-prompts-and-assistants Task 2.3 Register the test graph under a second graph ID](tasks.md#L14), a developer deploys the new `langgraph.json` once to find out, before anyone builds the fixture. +- Tests need a second workspace, a second deployment and one model provider key for the structured output check. codekiln creates the workspace and the key, and langstar's tests read them through `fnox.toml` and the CI workflow; the destination deployment fixture from [export-import-prompts-and-assistants Task 2.4 Add a destination deployment fixture](tasks.md#L19) creates the deployment whenever the cleanup workflow has deleted it, as the source fixture does today. Nobody knows yet whether the control plane accepts a second deployment from this repository. In [export-import-prompts-and-assistants Task 2.3 Register the test graph under a second graph ID](tasks.md#L18), a developer deploys the new `langgraph.json` once to find out, before anyone builds the fixture. - Docs: a new usage page on moving prompts and assistants between environments. ## Citations diff --git a/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md index 0816e4acd..4415c6067 100644 --- a/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md +++ b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md @@ -205,7 +205,7 @@ When the destination record holds a commit for the destination workspace and the ### Requirement: Read the prompt back after import -After writing, `langstar prompt import` SHALL record the commit hash in the destination record, then read the destination commit back with the model included and compare it with the file. When they differ it SHALL list the differences and exit with status 3, and the record SHALL keep the hash, so that the next import reports a change only when someone pushed a commit after the one langstar wrote. +After writing, `langstar prompt import` SHALL record the commit hash in the destination record, then read the destination commit back with the model included and compare it with the file as `langstar prompt compare` does. When they differ it SHALL list the differences and exit with status 3, and the record SHALL keep the hash, so that the next import reports a change only when someone pushed a commit after the one langstar wrote. #### Scenario: LangSmith drops a field @@ -215,7 +215,7 @@ After writing, `langstar prompt import` SHALL record the commit hash in the dest ### Requirement: Compare a prompt with its export file -`langstar prompt compare ` SHALL read the prompt with the file's handle from the workspace, with its model and repo details, and list each difference by its JSON path with both values. `--commit ` SHALL choose the commit, and without it langstar SHALL read the latest. It SHALL compare every field in the file's `repo` and `commit`, including every manifest key and its `id` arrays. Those hold no IDs or timestamps LangSmith sets, so compare never reports them. +`langstar prompt compare ` SHALL read the prompt with the file's handle from the workspace, with its model and repo details, and list each difference by its JSON path with both values. `--commit ` SHALL choose the commit, and without it the latest. It SHALL compare every field in the file's `repo` and `commit`, including every manifest key and its `id` arrays, except `commit.model_config` and `commit.model_provider`, which import cannot set. #### Scenario: Prompt matches @@ -232,6 +232,11 @@ After writing, `langstar prompt import` SHALL record the commit hash in the dest - **WHEN** someone changed the workspace's prompt to use `ChatAnthropic` where the file's manifest names `ChatOpenAI` - **THEN** langstar lists the JSON path of the model's `id` array with both values and exits with status 3 +#### Scenario: Destination fills the model configuration differently + +- **WHEN** the workspace's prompt has the file's manifest and a `model_config` or `model_provider` that differs from the file's +- **THEN** langstar reports no differences and exits with status 0 + #### Scenario: Repo readme changed - **WHEN** someone changed the readme of the workspace's prompt diff --git a/openspec/changes/export-import-prompts-and-assistants/tasks.md b/openspec/changes/export-import-prompts-and-assistants/tasks.md index 65d76227a..5a39273aa 100644 --- a/openspec/changes/export-import-prompts-and-assistants/tasks.md +++ b/openspec/changes/export-import-prompts-and-assistants/tasks.md @@ -5,12 +5,16 @@ Each numbered group below is one pull request, merged in the order listed, excep ## 1. Check the API round trip before building on it - [ ] 1.1 Extract the commit get, commit list (`GET /api/v1/commits/{owner}/{repo}`), commit create, repo update (`PATCH /api/v1/repos/{owner}/{repo}`), repo tags (`/api/v1/repos/{owner}/{repo}/tags` and `/api/v1/repos/{owner}/{repo}/tags/{tag_name}`), workspace secrets and playground settings endpoints from `reference/openapi/langchain/langsmith/openapi.json` into `reference/api-specs/langsmith/` and add their jq queries to `FRAGMENTS.md`; verify by rerunning each query and getting the same file -- [ ] 1.2 Write `reference/research/807-openapi-validation.md`, recording from the live API: the manifest returned with `include=model` for a structured prompt with a model, the outer `id` of that manifest and the `model_provider` value returned beside it, whether pushing that manifest to a second workspace returns the same model, what pushing an identical manifest returns, what the Agent Server returns for `context`, whether an Agent Server assistant update that sends a metadata key with a null value removes that key, whether an update that sends `config` or `context` replaces the assistant's object or merges into it, whether an update can clear an assistant's description, and what `POST /api/v1/prompts/invoke_prompt` returns for a prompt with a model; verify that every claim in the report names the request that produced it +- [ ] 1.2 Write `reference/research/807-openapi-validation.md`, recording from the live API: the manifest returned with `include=model` for a structured prompt with a model, the outer `id` of that manifest, the `model_config` and `model_provider` values returned beside it, whether pushing that manifest to a second workspace returns the same manifest, `model_config` and `model_provider`, whether `model_provider` matches the class name export writes to `requires.model_class`, what pushing an identical manifest returns, what the Agent Server returns for `context`, whether an Agent Server assistant update that sends a metadata key with a null value removes that key, whether an update that sends `config` or `context` replaces the assistant's object or merges into it, whether an update can clear an assistant's description, and what `POST /api/v1/prompts/invoke_prompt` returns for a prompt with a model; verify that every claim in the report names the request that produced it ## 2. Test environments - [ ] 2.1 Add the destination workspace and the model provider key to `fnox.toml`, the CI workflow and `docs/dev/environment-variables.md`, with placeholders only; verify with `mise run check` and a CI run that reads both. This task waits on codekiln to create a second workspace in the test organization and an OpenAI API key, and to store both in 1Password and the repository's CI secrets and the key as the destination workspace's `OPENAI_API_KEY` secret. The import tests need the workspace as a place to import into, and its secret lets prompt import's secret check pass for a prompt with an OpenAI model. The key lets the check in 7.3 run each imported structured prompt once against a small OpenAI model, which bills that account on every CI run; it is the one test that shows a copy still returns structured data, where every other test compares text. -- [ ] 2.2 Add `url` to `Deployment` in `sdk/src/deployments.rs` as an `Option` with `#[serde(default)]`, because the control-plane OpenAPI spec declares it nullable and a plain `String` would make a deployment list holding an unprovisioned deployment fail to parse, with a method that returns `url` and falls back to `source_config.custom_url` when `url` is null, missing or empty, and make `resolve_deployment_url` in `cli/src/deployment_utils.rs` call that method, so that a deployment built from GitHub resolves; end the four `Deployment` struct literals in `sdk/tests/integration_deployment_workflow.rs` and `cli/src/commands/deployment.rs` with `..Default::default()`, because a crate that builds a `Deployment` with a struct literal stops compiling when the field is added; add a unit test for each case in `sdk/src/deployments.rs`, among them a deployment whose `url` is null and one whose JSON has no `url` key, and verify by removing `#[ignore]` from `test_assistant_create_basic` in `cli/tests/assistant_command_test.rs` and running it against the test deployment +- [ ] 2.2 Make `resolve_deployment_url` in `cli/src/deployment_utils.rs` find the Agent Server URL of a deployment built from GitHub: + - Add `url` to `Deployment` in `sdk/src/deployments.rs` as an `Option` with `#[serde(default)]`. The control-plane OpenAPI spec declares `url` nullable, and a plain `String` would make a deployment list fail to parse when one deployment in it has no URL yet. + - Add a method to `Deployment` that returns `url`, or `source_config.custom_url` when `url` is null, missing or empty, and have `resolve_deployment_url` call it. + - End the four `Deployment` struct literals in `sdk/tests/integration_deployment_workflow.rs` and `cli/src/commands/deployment.rs` with `..Default::default()`, so they still compile after the new field lands. + - Verify with a unit test in `sdk/src/deployments.rs` for each case, among them a deployment whose `url` is null and one whose JSON has no `url` key, and by replacing the bare `#[ignore]` on `test_assistant_create_basic` in `cli/tests/assistant_command_test.rs` with `#[cfg_attr(not(feature = "integration-tests"), ignore)]`, the guard `test_assistant_description_field` beside it carries, and running it against the test deployment. - [ ] 2.3 Add a second `langgraph.json` under `tests/fixtures/` that registers the test graph from `tests/fixtures/test-graph-deployment/langgraph.json` under two graph IDs, `test_graph` and `test_graph_alt`, so the update tests in 3.3 and 10.2 can move an assistant from one graph to the other. To verify, deploy the new file while the source test deployment is live, which shows whether the control plane accepts a second deployment from this repository: run `langstar deployment create` with `--branch` set to this pull request's branch and `--config-path` set to the new file. A 409 "A deployment already exists for this agent environment" means it does not, and the developer reports that to codekiln before anyone starts task 2.4. Otherwise run `langstar assistant list --deployment `, find one assistant with graph ID `test_graph` and one with `test_graph_alt`, then delete the deployment. - [ ] 2.4 Add a destination deployment fixture to `sdk/src/test_utils.rs` and `cli/tests/common/fixtures.rs` that deploys the `langgraph.json` from 2.3 under the name prefix `dest-integration-test`. `find_reusable_deployment` returns any deployment whose name starts with the fixture's prefix before it checks the graph config, so this prefix keeps the source fixture, whose prefix is `pr-integration-test`, from picking up the destination deployment. `find_reusable_deployment` then falls back to any live deployment built from the same repository and graph config file, so the separate file from 2.3 keeps the destination fixture from picking up the source deployment. The control plane answers a second deployment of one agent environment with 409 "A deployment already exists for this agent environment". No document says which fields make an agent environment, and the verification in 2.3 shows whether a second graph config file in the same repository makes a new one. `.github/workflows/cleanup-test-deployments.yml` runs every four hours and deletes each deployment whose name contains `integration-test` once it is four hours old or older, so the destination deployment is deleted and rebuilt on the same schedule as the source deployment. Verify with an integration test that asserts the destination and source deployment IDs differ and lists assistants on the destination deployment through the URL that 2.2 resolves @@ -26,7 +30,10 @@ Each task in this group adds a public field to one of these SDK structs: `Prompt ## 4. Shared code for the prompt and assistant commands: the export file, the comparison and the report - [ ] 4.1 Add the export file types, with sorted keys, no export time and a reader that accepts only `format_version` 1 and the expected `kind`, and the destination record file, to a new module in `cli/src/`; verify with unit tests that build equivalent nested values with keys inserted in different orders, serialize each, and assert identical text with keys sorted at every level, and that updating one destination leaves the others unchanged, and that reading a file whose `format_version` is 2, or whose `kind` names the other resource, returns an error naming the field and its value -- [ ] 4.2 Add the JSON comparison, which compares every key under the `repo`, `commit` and `assistant` objects and reports differences by JSON Pointer; verify with unit tests on nested objects, arrays and missing keys, and a test that a changed `id` array inside `commit.manifest`, such as `ChatOpenAI` replaced by `ChatAnthropic`, is reported at its path +- [ ] 4.2 Add the JSON comparison, which compares every key under the `repo`, `commit` and `assistant` objects except `commit.model_config` and `commit.model_provider`, and reports differences by JSON Pointer. Verify with these tests: + - Unit tests on nested objects, arrays and missing keys. + - A test that a changed `id` array inside `commit.manifest`, such as `ChatOpenAI` replaced by `ChatAnthropic`, is reported at its path. + - A test that a destination whose `commit.model_config` and `commit.model_provider` differ from the file's, while the manifest matches, is reported as a match. - [ ] 4.3 Add the shared result report and the exit statuses 0, 3 and 1, printing JSON to stdout and progress to stderr, and leave clap's status 2 for usage errors; verify with unit tests of the report and of the exit status each result gives - [ ] 4.4 Add the secret reference walk that lists secret names from a manifest; verify with unit tests on a copy of the `prompt_with_model` manifest from the LangSmith Python SDK's integration tests, which `references/langsmith-sdk.md` links, and on a manifest with no secret reference @@ -50,7 +57,11 @@ Each task in this group adds a public field to one of these SDK structs: `Prompt - A CLI test that `prompt compare` with a mistyped flag exits with status 2. - A CLI test that `prompt compare` of a file holding `"format_version": 2` exits with status 1 and names `format_version` and the value 2. - A CLI test that `prompt compare` of a file whose handle the workspace lacks exits with status 1 and names the handle. -- [ ] 6.2 Add `langstar prompt verify` with the secret check and the saved model configuration note, matching `requires.model_class` against the class name at the end of each configuration's settings `id`; verify with integration tests against a workspace that lacks a listed secret and one that has it, a test that a missing saved model configuration is reported and leaves the exit status at 0, a test that verify of a prompt without a model, whose `requires.model_class` is null, reports no saved model configuration note and exits with status 0, and a test that a file whose manifest refers to a secret its `requires.secrets` leaves out exits with status 1 and names `requires.secrets` +- [ ] 6.2 Add `langstar prompt verify` with the secret check and the saved model configuration note, which matches `requires.model_class` against the class name at the end of each configuration's settings `id`. Verify with these tests: + - Integration tests against a workspace that lacks a secret the file lists, and against one that has it. + - A test that a missing saved model configuration is reported as a note and leaves the exit status at 0. + - A test that verify of a prompt without a model, whose `requires.model_class` is null, reports no saved model configuration note and exits with status 0. + - A test that a file whose manifest refers to a secret its `requires.secrets` leaves out exits with status 1 and names `requires.secrets`. - [ ] 6.3 Document compare and verify; verify that each documented command runs as written ## 7. prompt import @@ -80,6 +91,7 @@ Each task in this group adds a public field to one of these SDK structs: `Prompt - [ ] 8.1 Add `langstar assistant export` with `--output` and `--prompt-ref`. Verify with these tests: - An integration test that creates an assistant with configuration, context and metadata through the SDK and asserts every field of the exported file. - An integration test that exports, with `--prompt-ref /config/configurable/system_prompt`, an assistant whose value there is `support-system:prod`, and asserts that the file holds that pointer and `support-system:prod` under `requires.prompts`. + - An integration test that exports the same assistant version twice and asserts that the two files are identical. - An error test for a pointer that names nothing. - An error test that exporting an assistant ID the deployment lacks exits with status 1, names the ID and writes no file. - A CLI test that runs `assistant export` without `--output`, parses all of stdout as one JSON document, and asserts that its `source.assistant_id` is the exported assistant's ID and that any progress message is on stderr. @@ -92,7 +104,12 @@ Each task in this group adds a public field to one of these SDK structs: `Prompt - A unit test that compare reads the assistant ID from a destination record written by the 4.1 types. - An integration test that `assistant compare` with no destination record entry and no `--assistant-id` exits with status 1 and names the deployment. - A unit test that a file whose configuration names another prompt at a pointer under `requires.prompts` exits with status 1 and names the pointer and both values. -- [ ] 9.2 Add `langstar assistant verify` with `--assistant-id`, `--workspace-id` for the workspace that holds the prompts, the graph check, and the prompt check, which reads `GET /api/v1/commits/{owner}/{repo}/{tag}`; verify with integration tests that run verify with `--assistant-id` against an exported assistant, one with a file whose graph ID the deployment lacks, one with a missing prompt tag, and one that passes `--workspace-id` set to the destination workspace, and a CLI test that `langstar -f json assistant verify` prints one JSON object with `status`, `differences` and `checks` to stdout +- [ ] 9.2 Add `langstar assistant verify` with `--assistant-id`, `--workspace-id` for the workspace that holds the prompts, the graph check, and the prompt check, which reads `GET /api/v1/commits/{owner}/{repo}/{tag}`. Verify with these tests: + - An integration test that runs verify with `--assistant-id` against an exported assistant and exits with status 0. + - An integration test of a file whose graph ID the deployment lacks, which reports the graph ID and exits with status 3. + - An integration test of a file whose prompt tag the workspace lacks, which reports the prompt and exits with status 3. + - An integration test that passes `--workspace-id` set to the destination workspace. + - A CLI test that `langstar -f json assistant verify` prints one JSON object with `status`, `differences` and `checks` to stdout. - [ ] 9.3 Document assistant compare and verify; verify that each documented command runs as written ## 10. assistant import From bd440fcf4b243128a918b03b83ab1c08a4e064d5 Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Wed, 14 Oct 2026 00:07:42 -0400 Subject: [PATCH 41/90] =?UTF-8?q?=F0=9F=93=9A=20docs(openspec):=20replace?= =?UTF-8?q?=20the=20destination=20record=20by=20rename,=20drop=20source.re?= =?UTF-8?q?f=20from=20the=20export=20example,=20and=20warn=20that=20assist?= =?UTF-8?q?ant=20export=20copies=20credentials=20kept=20in=20configuration?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit #807 Export, import, compare, and verify prompts and assistants across LangSmith environments Co-Authored-By: Claude Opus 5.5 --- .../export-import-prompts-and-assistants/design.md | 5 +++-- .../specs/assistant/spec.md | 10 ++++++++++ .../export-import-prompts-and-assistants/tasks.md | 8 ++++++-- 3 files changed, 19 insertions(+), 4 deletions(-) diff --git a/openspec/changes/export-import-prompts-and-assistants/design.md b/openspec/changes/export-import-prompts-and-assistants/design.md index 1488c81de..2738c5a13 100644 --- a/openspec/changes/export-import-prompts-and-assistants/design.md +++ b/openspec/changes/export-import-prompts-and-assistants/design.md @@ -40,7 +40,7 @@ One prompt export looks like this: { "kind": "prompt", "format_version": 1, - "source": { "workspace_id": "", "handle": "my-prompt", "commit_hash": "", "ref": "prod" }, + "source": { "workspace_id": "", "handle": "my-prompt", "commit_hash": "" }, "repo": { "handle": "my-prompt", "description": "...", "readme": "...", "tags": ["support"], "is_public": false }, "commit": { "description": "...", "manifest": { "lc": 1, "type": "constructor", "id": ["langsmith", "playground", "PromptPlayground"], "kwargs": { "first": {}, "last": {} } }, "model_config": {}, "model_provider": "...", "tags": ["prod"] }, "requires": { "secrets": ["OPENAI_API_KEY"], "model_class": "ChatOpenAI" } @@ -61,7 +61,7 @@ Export reads the commit with `include=model`, the form the Go SDK uses (`referen ### Import records what it created in a separate file -Import writes the IDs of what it created to `.destinations.json`, keyed by destination workspace ID for prompts and by deployment ID for assistants. The export file stays the same after any number of imports, and a fresh export never wipes out the record of earlier imports. +Import writes the IDs of what it created to `.destinations.json`, keyed by destination workspace ID for prompts and by deployment ID for assistants. The export file stays the same after any number of imports, and a fresh export never wipes out the record of earlier imports. Import replaces the record by writing the whole new record to a temporary file in the same directory and renaming it over the old one, so a langstar that stops in the middle of a write leaves the earlier record whole, pending assistant IDs included. A repeated prompt import finds the copy by its handle and uses the record to tell whether someone changed it. Import writes the new commit hash to the record before it reads the commit back, so a read-back that finds a difference still leaves the hash in the record. The next import compares the destination's latest commit with that hash, and reports a change only when someone else pushed a commit after langstar's. @@ -121,6 +121,7 @@ A mise task runs a short Python script that pulls the imported prompt with `incl - [Pushing a manifest identical to the latest commit may return 409] → Import compares first and pushes nothing when the prompt matches. - [langstar finds a deployment's Agent Server URL only in `source_config.custom_url` (`resolve_deployment_url` in `cli/src/deployment_utils.rs`), and a deployment built from GitHub has none, so every `langstar assistant` command, the new ones included, fails on such a deployment] → In [export-import-prompts-and-assistants Task 2.2 Return the URL of a deployment built from GitHub](tasks.md#L13), a developer adds the control plane's top-level `url` to the SDK's `Deployment`, which drops it today, and makes `resolve_deployment_url` use it, before anyone builds the assistant export, compare, verify and import commands. - [The SDK changes for export and import add public fields to `Deployment`, `Prompt`, `CommitManifestResponse`, `CommitData`, `CommitRequest`, `Assistant` and the assistant request structs, so a crate that builds one of those structs with a struct literal or destructures it with a pattern that names every field stops compiling] → Each pull request that adds a field updates every struct literal of that struct in this repository. [Should the SDK field additions count as a breaking change?](#9---should-the-sdk-field-additions-count-as-a-breaking-change) asks codekiln whether those pull requests also carry a `BREAKING CHANGE:` footer, which `docs/dev/git-scm-conventions.md` says makes the release a major version. +- [An assistant's configuration, context or metadata may hold a credential, which export copies into a file meant for Git] → Langstar cannot tell a credential from any other string in an assistant's configuration, as it cannot tell which key holds a prompt name, so export writes those objects as the Agent Server returns them. The assistant export docs in [export-import-prompts-and-assistants Task 8.2 Document assistant export](tasks.md#L102) tell users to keep credentials in the deployment's environment variables or the workspace's secrets rather than in an assistant, and to read an export file before committing it. - [A tag in the destination may already point at another commit] → `--update` moves the tag, and `--dry-run` lists the move. - [`--update` deletes a tag on the matching destination commit that the file lacks, and a graph on a deployment may read that tag] → `--dry-run` lists each delete, and `assistant verify` reports a prompt tag an assistant refers to that the workspace no longer has. - [The control plane may refuse a second deployment from this repository with 409] → In [export-import-prompts-and-assistants Task 2.3 Register the test graph under a second graph ID](tasks.md#L18), a developer deploys the new `langgraph.json` once, before anyone builds the fixture. If the control plane answers with 409, the developer brings [Which second workspace and second deployment should the tests use?](#5---which-second-workspace-and-second-deployment-should-the-tests-use) back to codekiln. diff --git a/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md b/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md index 228dd127e..b6b9cf907 100644 --- a/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md +++ b/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md @@ -83,6 +83,16 @@ Every `langstar assistant` subcommand SHALL take a required `--deployment` flag - **THEN** langstar sends the create with that ID and `if_exists` set to `do_nothing`, so the deployment still holds one assistant from the file - **AND** the destination record marks that ID created +### Requirement: Replace the destination record in one step + +`langstar assistant import` SHALL write each new destination record to a temporary file in the record's directory and rename it over the old record, so that a langstar that stops during a write leaves the earlier record whole, with its pending assistant IDs. + +#### Scenario: Import stopped while marking the ID created + +- **WHEN** langstar stops while it writes the record that marks a new assistant ID created +- **THEN** the destination record still holds that ID marked pending +- **AND** the next `langstar assistant import support-bot.json --deployment my-agent-prod` sends the create with that ID and creates no second assistant + ### Requirement: Check the destination before importing an assistant Before it writes, `langstar assistant import` SHALL run the graph check and the prompt check of `assistant verify`, reading prompts from `--workspace-id` or the configured workspace. When a check fails it SHALL write nothing, name each missing graph and prompt, and exit with status 3. diff --git a/openspec/changes/export-import-prompts-and-assistants/tasks.md b/openspec/changes/export-import-prompts-and-assistants/tasks.md index 5a39273aa..b592cdbc2 100644 --- a/openspec/changes/export-import-prompts-and-assistants/tasks.md +++ b/openspec/changes/export-import-prompts-and-assistants/tasks.md @@ -29,7 +29,11 @@ Each task in this group adds a public field to one of these SDK structs: `Prompt ## 4. Shared code for the prompt and assistant commands: the export file, the comparison and the report -- [ ] 4.1 Add the export file types, with sorted keys, no export time and a reader that accepts only `format_version` 1 and the expected `kind`, and the destination record file, to a new module in `cli/src/`; verify with unit tests that build equivalent nested values with keys inserted in different orders, serialize each, and assert identical text with keys sorted at every level, and that updating one destination leaves the others unchanged, and that reading a file whose `format_version` is 2, or whose `kind` names the other resource, returns an error naming the field and its value +- [ ] 4.1 Add the export file types, with sorted keys, no export time and a reader that accepts only `format_version` 1 and the expected `kind`, and the destination record file, which langstar replaces by writing a temporary file in the same directory and renaming it over the old one, to a new module in `cli/src/`. Verify with these unit tests: + - Tests that build equivalent nested values with keys inserted in different orders, serialize each, and assert identical text with keys sorted at every level. + - A test that updating one destination leaves the others unchanged. + - A test that a record write that fails before the rename leaves the earlier record file unchanged, its pending entries included. + - A test that reading a file whose `format_version` is 2, or whose `kind` names the other resource, returns an error naming the field and its value. - [ ] 4.2 Add the JSON comparison, which compares every key under the `repo`, `commit` and `assistant` objects except `commit.model_config` and `commit.model_provider`, and reports differences by JSON Pointer. Verify with these tests: - Unit tests on nested objects, arrays and missing keys. - A test that a changed `id` array inside `commit.manifest`, such as `ChatOpenAI` replaced by `ChatAnthropic`, is reported at its path. @@ -95,7 +99,7 @@ Each task in this group adds a public field to one of these SDK structs: `Prompt - An error test for a pointer that names nothing. - An error test that exporting an assistant ID the deployment lacks exits with status 1, names the ID and writes no file. - A CLI test that runs `assistant export` without `--output`, parses all of stdout as one JSON document, and asserts that its `source.assistant_id` is the exported assistant's ID and that any progress message is on stderr. -- [ ] 8.2 Document assistant export; verify that each documented command runs as written +- [ ] 8.2 Document assistant export, including that export writes the assistant's configuration, context and metadata as the Agent Server returns them, so a credential kept there reaches the file; verify that each documented command runs as written ## 9. assistant compare and verify From 4e2c2a2c34a4045bc79fdfa969a9d4838e0a9f43 Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Wed, 14 Oct 2026 00:20:36 -0400 Subject: [PATCH 42/90] =?UTF-8?q?=F0=9F=93=9A=20docs(openspec):=20lead=20t?= =?UTF-8?q?he=20compare=20sentence=20with=20what=20compare=20checks,=20mov?= =?UTF-8?q?e=20the=20model=5Fprovider=20citation=20to=20export,=20require?= =?UTF-8?q?=20the=20prompt=20record=20rename,=20and=20spell=20out=20the=20?= =?UTF-8?q?assistant=20export=20docs=20advice?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit #807 Export, import, compare, and verify prompts and assistants across LangSmith environments Co-Authored-By: Claude Opus 5.5 --- .../export-import-prompts-and-assistants/design.md | 6 +++--- .../specs/prompt/spec.md | 10 ++++++++++ .../export-import-prompts-and-assistants/tasks.md | 2 +- 3 files changed, 14 insertions(+), 4 deletions(-) diff --git a/openspec/changes/export-import-prompts-and-assistants/design.md b/openspec/changes/export-import-prompts-and-assistants/design.md index 2738c5a13..9a86bb3bc 100644 --- a/openspec/changes/export-import-prompts-and-assistants/design.md +++ b/openspec/changes/export-import-prompts-and-assistants/design.md @@ -57,7 +57,7 @@ Import, compare and verify read a file only when its `format_version` is 1 and i ### Export asks LangSmith for the model -Export reads the commit with `include=model`, the form the Go SDK uses (`references/langsmith-go.md`). The Python SDK sends the older `include_model=true`, and the LangSmith OpenAPI spec accepts both. With the model included, the manifest wraps the prompt and the chat model with its settings in one constructor, and the response also carries `model_config` and `model_provider` beside the manifest (`references/langsmith-sdk.md`). No field of the create-commit request holds the model, so the model reaches the destination inside the manifest, and export keeps `model_config` and `model_provider` in the file only for the reader. In [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8), a developer pushes that manifest to a second workspace and records whether LangSmith returns the same manifest, `model_config` and `model_provider`, and which outer `id` the live API returns. +Export reads the commit with `include=model`, the form the Go SDK uses (`references/langsmith-go.md`). The Python SDK sends the older `include_model=true`, and the LangSmith OpenAPI spec accepts both. With the model included, the manifest wraps the prompt and the chat model with its settings in one constructor, and the response also carries `model_config` and `model_provider` beside the manifest (`references/langsmith-sdk.md`). No field of the create-commit request holds the model, so the model reaches the destination inside the manifest, and export keeps `model_config` and `model_provider` in the file only for the reader. The LangSmith Python SDK's [`PromptCommit`](https://github.com/langchain-ai/langsmith-sdk/blob/47c553caf08c1989e2be3ff1886da3eb801f709f/python/langsmith/schemas.py#L1054) documents `model_provider` as the model's class name, such as `ChatOpenAI`. In [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8), a developer pushes that manifest to a second workspace and records whether LangSmith returns the same manifest, `model_config` and `model_provider`, and which outer `id` the live API returns. ### Import records what it created in a separate file @@ -81,7 +81,7 @@ Without `--update`, import lists the differences and exits with status 3. `--dry ### Compare reports differences by JSON path -Compare reads the prompt or assistant from the destination, puts it in the shape of the export file's `repo`, `commit` and `assistant` objects, and lists every value that differs from the file. Apart from `commit.model_config` and `commit.model_provider`, which the next paragraph covers, those objects hold no field the server sets, such as IDs, version numbers, timestamps, owner and like counts, so compare checks every other key in them, including the `id` arrays inside a manifest that name the prompt and model classes, such as `ChatOpenAI`. The source environment's workspace, deployment and IDs sit in `source`, so they never show up as differences against a copy. Each difference names its JSON Pointer path and both values. For a prompt, the comparison covers the commit description, the commit tags and the commit each one points at, any tag on the compared commit that the file lacks, and the repo's description, readme, tags and visibility as well as the manifest. Import runs the same comparison after it writes, so a field LangSmith or the Agent Server drops shows up as a difference. +Compare reads the prompt or assistant from the destination, puts it in the shape of the export file's `repo`, `commit` and `assistant` objects, and lists every value that differs from the file. Compare checks every key in those objects, including the `id` arrays inside a manifest that name the prompt and model classes, such as `ChatOpenAI`, and skips `commit.model_config` and `commit.model_provider`, which LangSmith fills in for each workspace. Export leaves the values the server sets, such as IDs, version numbers, timestamps, owner and like counts, out of those objects, so they never show up as differences. The source environment's workspace, deployment and IDs sit in `source`, so they never show up as differences against a copy. Each difference names its JSON Pointer path and both values. For a prompt, the comparison covers the commit description, the commit tags and the commit each one points at, any tag on the compared commit that the file lacks, and the repo's description, readme, tags and visibility as well as the manifest. Import runs the same comparison after it writes, so a field LangSmith or the Agent Server drops shows up as a difference. Compare and import's read-back leave out `commit.model_config` and `commit.model_provider`, because import cannot set them. No field of the create-commit request holds either one, so the destination fills them itself, and if it filled them differently from the source, every prompt import's read-back would report a difference and exit with status 3. The model reaches the destination inside the manifest, so compare checks the model there, through the chat model constructor's `id` array and settings. @@ -89,7 +89,7 @@ Compare and import's read-back leave out `commit.model_config` and `commit.model Prompt verify checks that the workspace holds every secret the prompt's model reads. Export lists those names in `requires.secrets` by walking the manifest for LangChain secret references, `{"lc": 1, "type": "secret", "id": ["OPENAI_API_KEY"]}`, the shape the Python SDK's `_manifest_has_secrets` matches. Verify looks each name up among the workspace's secrets (`GET /api/v1/workspaces/current/secrets`, which `langstar secrets list` already calls) and prints the name of each missing secret and no secret value. -Prompt verify also notes when the workspace has no saved model configuration for the prompt's model, and leaves the exit status alone, because the prompt runs with the model its manifest carries. Export takes the model's class name, such as `ChatOpenAI`, from the last element of the chat model constructor's `id` in the manifest, such as `["langchain", "chat_models", "openai", "ChatOpenAI"]`, and stores it in `requires.model_class`. Verify looks for a saved model configuration (`GET /api/v1/playground-settings`, which `langstar model-config list` calls) whose settings `id` ends in that class name. LangSmith also returns a `model_provider` field beside the manifest, which the file keeps as `commit.model_provider`. The LangSmith Python SDK's [`PromptCommit`](https://github.com/langchain-ai/langsmith-sdk/blob/47c553caf08c1989e2be3ff1886da3eb801f709f/python/langsmith/schemas.py#L1054) calls it the model provider and gives `ChatOpenAI` as its example, which is the class name export writes to `requires.model_class`. In [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8), a developer records the value the live API returns and whether it matches `requires.model_class`. [#807 Export, import, compare, and verify prompts and assistants across LangSmith environments](https://github.com/codekiln/langstar/issues/807) asks for the note, and [Should prompt verify report a missing saved model configuration?](#7---should-prompt-verify-report-a-missing-saved-model-configuration) asks whether to keep it. A prompt saved without a model has no chat model constructor in its manifest, so export writes `requires.model_class` as null, and verify and import skip the saved model configuration check for it and still run the secret check. +Prompt verify also notes when the workspace has no saved model configuration for the prompt's model, and leaves the exit status alone, because the prompt runs with the model its manifest carries. Export takes the model's class name, such as `ChatOpenAI`, from the last element of the chat model constructor's `id` in the manifest, such as `["langchain", "chat_models", "openai", "ChatOpenAI"]`, and stores it in `requires.model_class`. Verify looks for a saved model configuration (`GET /api/v1/playground-settings`, which `langstar model-config list` calls) whose settings `id` ends in that class name. A prompt saved without a model has no chat model constructor in its manifest, so export writes `requires.model_class` as null, and verify and import skip this check for it and still run the secret check. [#807 Export, import, compare, and verify prompts and assistants across LangSmith environments](https://github.com/codekiln/langstar/issues/807) asks for the note, and [Should prompt verify report a missing saved model configuration?](#7---should-prompt-verify-report-a-missing-saved-model-configuration) asks whether to keep it. Import and verify work out `requires` again from the file's manifest: the secret names, and the model class or null. When either differs from what the file holds, they write nothing, name the field and exit with status 1. This catches a manifest someone edited in Git without updating `requires`. diff --git a/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md index 4415c6067..a8efb2eef 100644 --- a/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md +++ b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md @@ -183,6 +183,16 @@ After import writes a commit, langstar SHALL record the destination workspace, h - **THEN** `my-prompt.destinations.json` holds one entry for each workspace with its own commit hash - **AND** `my-prompt.json` is unchanged +### Requirement: Replace the prompt destination record in one step + +`langstar prompt import` SHALL write each new destination record to a temporary file in the record's directory and rename it over the old record, so that a langstar that stops during a write leaves the earlier record whole. + +#### Scenario: Import stopped while writing the record + +- **WHEN** `my-prompt.destinations.json` holds commit hashes for a staging and a production workspace +- **AND** langstar stops while it writes the record after an import into the production workspace +- **THEN** `my-prompt.destinations.json` still holds both workspaces' entries as they were before that import + ### Requirement: Warn when the destination changed since the last import When the destination record holds a commit for the destination workspace and the destination's latest commit is a different one, `langstar prompt import` SHALL say that someone changed the prompt after the last import, in addition to listing the differences. diff --git a/openspec/changes/export-import-prompts-and-assistants/tasks.md b/openspec/changes/export-import-prompts-and-assistants/tasks.md index b592cdbc2..4f5d08080 100644 --- a/openspec/changes/export-import-prompts-and-assistants/tasks.md +++ b/openspec/changes/export-import-prompts-and-assistants/tasks.md @@ -99,7 +99,7 @@ Each task in this group adds a public field to one of these SDK structs: `Prompt - An error test for a pointer that names nothing. - An error test that exporting an assistant ID the deployment lacks exits with status 1, names the ID and writes no file. - A CLI test that runs `assistant export` without `--output`, parses all of stdout as one JSON document, and asserts that its `source.assistant_id` is the exported assistant's ID and that any progress message is on stderr. -- [ ] 8.2 Document assistant export, including that export writes the assistant's configuration, context and metadata as the Agent Server returns them, so a credential kept there reaches the file; verify that each documented command runs as written +- [ ] 8.2 Document assistant export. The page says that export writes the assistant's configuration, context and metadata as the Agent Server returns them, so a credential kept there reaches the file; tells users to keep credentials in the deployment's environment variables or the workspace's secrets; and tells them to read an export file before they commit it. Verify that each documented command runs as written. ## 9. assistant compare and verify From 0ca0d29931ccb32dbcbe59a8aee1c25bb1b406d7 Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Wed, 14 Oct 2026 00:32:44 -0400 Subject: [PATCH 43/90] =?UTF-8?q?=F0=9F=93=9A=20docs(openspec):=20record?= =?UTF-8?q?=20a=20matching=20commit=20an=20interrupted=20import=20pushed,?= =?UTF-8?q?=20ask=20question=201=20about=20status=203,=20and=20give=20the?= =?UTF-8?q?=206.2=20and=209.2=20tests=20their=20assertions?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit #807 Export, import, compare, and verify prompts and assistants across LangSmith environments Co-Authored-By: Claude Opus 5.5 --- .../design.md | 8 ++++---- .../specs/prompt/spec.md | 18 +++++++++++++++++- .../tasks.md | 6 ++++-- 3 files changed, 25 insertions(+), 7 deletions(-) diff --git a/openspec/changes/export-import-prompts-and-assistants/design.md b/openspec/changes/export-import-prompts-and-assistants/design.md index 9a86bb3bc..22dcf55bb 100644 --- a/openspec/changes/export-import-prompts-and-assistants/design.md +++ b/openspec/changes/export-import-prompts-and-assistants/design.md @@ -63,7 +63,7 @@ Export reads the commit with `include=model`, the form the Go SDK uses (`referen Import writes the IDs of what it created to `.destinations.json`, keyed by destination workspace ID for prompts and by deployment ID for assistants. The export file stays the same after any number of imports, and a fresh export never wipes out the record of earlier imports. Import replaces the record by writing the whole new record to a temporary file in the same directory and renaming it over the old one, so a langstar that stops in the middle of a write leaves the earlier record whole, pending assistant IDs included. -A repeated prompt import finds the copy by its handle and uses the record to tell whether someone changed it. Import writes the new commit hash to the record before it reads the commit back, so a read-back that finds a difference still leaves the hash in the record. The next import compares the destination's latest commit with that hash, and reports a change only when someone else pushed a commit after langstar's. +A repeated prompt import finds the copy by its handle and uses the record to tell whether someone changed it. Import writes the new commit hash to the record before it reads the commit back, so a read-back that finds a difference still leaves the hash in the record. The next import compares the destination's latest commit with that hash, and reports a change only when someone else pushed a commit after langstar's. An import that stopped after its push and before its record write leaves the earlier hash, or no entry, in the record, so when the destination's latest commit matches the file, the next import writes that commit's hash to the record and reports no change. An assistant's ID differs on every deployment, so the record is the only way a repeated import finds the assistant it made. Import chooses the new assistant's ID itself and writes it to the record, marked pending, before it sends the create request with that `assistant_id` and `if_exists` set to `do_nothing`, which the Agent Server's `AssistantCreate` accepts. After the create succeeds, import marks the entry created. If langstar stops before then, the next import finds the pending entry and sends the same create again, and the Agent Server returns the assistant the first import made. @@ -83,7 +83,7 @@ Without `--update`, import lists the differences and exits with status 3. `--dry Compare reads the prompt or assistant from the destination, puts it in the shape of the export file's `repo`, `commit` and `assistant` objects, and lists every value that differs from the file. Compare checks every key in those objects, including the `id` arrays inside a manifest that name the prompt and model classes, such as `ChatOpenAI`, and skips `commit.model_config` and `commit.model_provider`, which LangSmith fills in for each workspace. Export leaves the values the server sets, such as IDs, version numbers, timestamps, owner and like counts, out of those objects, so they never show up as differences. The source environment's workspace, deployment and IDs sit in `source`, so they never show up as differences against a copy. Each difference names its JSON Pointer path and both values. For a prompt, the comparison covers the commit description, the commit tags and the commit each one points at, any tag on the compared commit that the file lacks, and the repo's description, readme, tags and visibility as well as the manifest. Import runs the same comparison after it writes, so a field LangSmith or the Agent Server drops shows up as a difference. -Compare and import's read-back leave out `commit.model_config` and `commit.model_provider`, because import cannot set them. No field of the create-commit request holds either one, so the destination fills them itself, and if it filled them differently from the source, every prompt import's read-back would report a difference and exit with status 3. The model reaches the destination inside the manifest, so compare checks the model there, through the chat model constructor's `id` array and settings. +LangSmith fills in `model_config` and `model_provider` itself when import pushes a commit, since no field of the create-commit request holds either one. When the destination fills them in differently from the source, a check on them would fail every prompt import with status 3. Compare checks the model inside the manifest instead, through the chat model constructor's `id` array and its settings. ### Verify checks what the copy needs to run @@ -121,7 +121,7 @@ A mise task runs a short Python script that pulls the imported prompt with `incl - [Pushing a manifest identical to the latest commit may return 409] → Import compares first and pushes nothing when the prompt matches. - [langstar finds a deployment's Agent Server URL only in `source_config.custom_url` (`resolve_deployment_url` in `cli/src/deployment_utils.rs`), and a deployment built from GitHub has none, so every `langstar assistant` command, the new ones included, fails on such a deployment] → In [export-import-prompts-and-assistants Task 2.2 Return the URL of a deployment built from GitHub](tasks.md#L13), a developer adds the control plane's top-level `url` to the SDK's `Deployment`, which drops it today, and makes `resolve_deployment_url` use it, before anyone builds the assistant export, compare, verify and import commands. - [The SDK changes for export and import add public fields to `Deployment`, `Prompt`, `CommitManifestResponse`, `CommitData`, `CommitRequest`, `Assistant` and the assistant request structs, so a crate that builds one of those structs with a struct literal or destructures it with a pattern that names every field stops compiling] → Each pull request that adds a field updates every struct literal of that struct in this repository. [Should the SDK field additions count as a breaking change?](#9---should-the-sdk-field-additions-count-as-a-breaking-change) asks codekiln whether those pull requests also carry a `BREAKING CHANGE:` footer, which `docs/dev/git-scm-conventions.md` says makes the release a major version. -- [An assistant's configuration, context or metadata may hold a credential, which export copies into a file meant for Git] → Langstar cannot tell a credential from any other string in an assistant's configuration, as it cannot tell which key holds a prompt name, so export writes those objects as the Agent Server returns them. The assistant export docs in [export-import-prompts-and-assistants Task 8.2 Document assistant export](tasks.md#L102) tell users to keep credentials in the deployment's environment variables or the workspace's secrets rather than in an assistant, and to read an export file before committing it. +- [An assistant's configuration, context or metadata may hold a credential, which export copies into a file meant for Git] → Langstar cannot tell a credential from any other string in an assistant's configuration, as it cannot tell which key holds a prompt name, so export writes those objects as the Agent Server returns them. The assistant export docs in [export-import-prompts-and-assistants Task 8.2 Document assistant export](tasks.md#L104) tell users to keep credentials in the deployment's environment variables or the workspace's secrets rather than in an assistant, and to read an export file before committing it. - [A tag in the destination may already point at another commit] → `--update` moves the tag, and `--dry-run` lists the move. - [`--update` deletes a tag on the matching destination commit that the file lacks, and a graph on a deployment may read that tag] → `--dry-run` lists each delete, and `assistant verify` reports a prompt tag an assistant refers to that the workspace no longer has. - [The control plane may refuse a second deployment from this repository with 409] → In [export-import-prompts-and-assistants Task 2.3 Register the test graph under a second graph ID](tasks.md#L18), a developer deploys the new `langgraph.json` once, before anyone builds the fixture. If the control plane answers with 409, the developer brings [Which second workspace and second deployment should the tests use?](#5---which-second-workspace-and-second-deployment-should-the-tests-use) back to codekiln. @@ -132,7 +132,7 @@ None. ## Open Questions -### 1 - Should compare, verify and import exit with 2 when they find a difference? +### 1 - Should compare, verify and import exit with 3 when they find a difference? > Context from the drafting agent, for question 1. > Status 2 is taken: clap exits with 2 when a user mistypes a flag or leaves out a required one, so a difference reported as 2 would look like a typo to a script. The draft uses 0 for a match, 3 for a difference or failed check and 1 for every other error, and leaves clap's 2 alone. The other choice makes `run` in `cli/src/main.rs` exit with 1 on a usage error, which frees 2 for differences but changes the status every langstar command gives for a mistyped flag. `diff` and `git diff --exit-code` use 1 for differences, which would make a difference look like an error. Recommendation: 3 for a difference. diff --git a/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md index a8efb2eef..de6283e8b 100644 --- a/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md +++ b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md @@ -195,7 +195,7 @@ After import writes a commit, langstar SHALL record the destination workspace, h ### Requirement: Warn when the destination changed since the last import -When the destination record holds a commit for the destination workspace and the destination's latest commit is a different one, `langstar prompt import` SHALL say that someone changed the prompt after the last import, in addition to listing the differences. +When the destination record holds a commit for the destination workspace and the destination's latest commit is a different one that also differs from the file, `langstar prompt import` SHALL say that someone changed the prompt after the last import, in addition to listing the differences. #### Scenario: Someone edited the destination prompt @@ -203,6 +203,22 @@ When the destination record holds a commit for the destination workspace and the - **AND** runs `langstar prompt import my-prompt.json --workspace-id ` - **THEN** langstar reports that the destination changed since the last recorded import and exits with status 3 +### Requirement: Record a matching commit that an earlier import pushed + +When the destination's latest commit matches the file and the destination record holds no commit for the destination workspace or a different one, `langstar prompt import` without `--dry-run` SHALL write that commit's hash to the record, whatever hash the record held, and SHALL NOT say that someone changed the prompt. When the commit tags match the file too, it SHALL report the prompt unchanged and exit with status 0. + +#### Scenario: Earlier import stopped before it wrote the record + +- **WHEN** an earlier import pushed a commit and stopped before it wrote that commit's hash to `my-prompt.destinations.json` +- **AND** the user runs `langstar prompt import my-prompt.json --workspace-id ` again +- **THEN** langstar reports that the prompt is unchanged, writes the latest commit's hash to the record and exits with status 0 + +#### Scenario: First import stopped before it wrote the record + +- **WHEN** the first import into a workspace pushed the prompt and stopped before it wrote `my-prompt.destinations.json` +- **AND** the user runs `langstar prompt import my-prompt.json --workspace-id ` again +- **THEN** langstar reports that the prompt is unchanged, writes the latest commit's hash to the record under that workspace and exits with status 0 + ### Requirement: Dry run an import `langstar prompt import --dry-run` SHALL report whether it would create the prompt, add a commit, or leave it unchanged, list the differences, and change nothing in the workspace or the destination record. diff --git a/openspec/changes/export-import-prompts-and-assistants/tasks.md b/openspec/changes/export-import-prompts-and-assistants/tasks.md index 4f5d08080..97af913cf 100644 --- a/openspec/changes/export-import-prompts-and-assistants/tasks.md +++ b/openspec/changes/export-import-prompts-and-assistants/tasks.md @@ -62,7 +62,7 @@ Each task in this group adds a public field to one of these SDK structs: `Prompt - A CLI test that `prompt compare` of a file holding `"format_version": 2` exits with status 1 and names `format_version` and the value 2. - A CLI test that `prompt compare` of a file whose handle the workspace lacks exits with status 1 and names the handle. - [ ] 6.2 Add `langstar prompt verify` with the secret check and the saved model configuration note, which matches `requires.model_class` against the class name at the end of each configuration's settings `id`. Verify with these tests: - - Integration tests against a workspace that lacks a secret the file lists, and against one that has it. + - An integration test against a workspace that lacks a secret the file lists, asserting that verify names the secret and exits with status 3, and one against a workspace that has every listed secret, asserting that verify exits with status 0. - A test that a missing saved model configuration is reported as a note and leaves the exit status at 0. - A test that verify of a prompt without a model, whose `requires.model_class` is null, reports no saved model configuration note and exits with status 0. - A test that a file whose manifest refers to a secret its `requires.secrets` leaves out exits with status 1 and names `requires.secrets`. @@ -81,6 +81,8 @@ Each task in this group adds a public field to one of these SDK structs: `Prompt - An integration test that a repeated import creates no commit. - An integration test that a changed destination is refused without `--update`. - An integration test that an import after someone pushed a commit to the destination since the recorded import says the destination changed since the last import and exits with status 3. + - An integration test that pushes the file's manifest to the destination, leaves an older commit hash in the destination record, imports the file, and asserts that import exits with status 0, prints no change warning and leaves the latest commit's hash in the record. + - An integration test that pushes the file's manifest and tags to the destination, imports the file with no destination record, and asserts that import exits with status 0 and writes the latest commit's hash to the record under the destination workspace. - An integration test that `--update` adds one commit with the right parent and the file's commit description. - An integration test that `--update` against a destination whose repo description, readme, tags or visibility differ while the manifest matches updates the repo and adds no commit. - An integration test that `--update` against a destination whose `prod` tag points at an earlier commit while the manifest matches moves the tag and adds no commit, and that `--dry-run` lists the move. @@ -112,7 +114,7 @@ Each task in this group adds a public field to one of these SDK structs: `Prompt - An integration test that runs verify with `--assistant-id` against an exported assistant and exits with status 0. - An integration test of a file whose graph ID the deployment lacks, which reports the graph ID and exits with status 3. - An integration test of a file whose prompt tag the workspace lacks, which reports the prompt and exits with status 3. - - An integration test that passes `--workspace-id` set to the destination workspace. + - An integration test that runs verify with `--workspace-id` set to the destination workspace, for an assistant whose referenced prompt tag exists only in that workspace, and asserts that verify exits with status 0 and that `checks` reports the prompt as found. - A CLI test that `langstar -f json assistant verify` prints one JSON object with `status`, `differences` and `checks` to stdout. - [ ] 9.3 Document assistant compare and verify; verify that each documented command runs as written From 7885b87d15518335326085b8cfe79470263a75a5 Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Wed, 14 Oct 2026 00:38:32 -0400 Subject: [PATCH 44/90] =?UTF-8?q?=F0=9F=93=9A=20docs(openspec):=20promise?= =?UTF-8?q?=20only=20that=20prompt=20export=20writes=20no=20value=20of=20a?= =?UTF-8?q?=20referenced=20secret,=20and=20warn=20in=20the=20prompt=20expo?= =?UTF-8?q?rt=20docs=20about=20credentials=20written=20as=20literal=20sett?= =?UTF-8?q?ings?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit #807 Export, import, compare, and verify prompts and assistants across LangSmith environments Co-Authored-By: Claude Opus 5.5 --- openspec/changes/export-import-prompts-and-assistants/design.md | 1 + .../export-import-prompts-and-assistants/specs/prompt/spec.md | 2 +- openspec/changes/export-import-prompts-and-assistants/tasks.md | 2 +- 3 files changed, 3 insertions(+), 2 deletions(-) diff --git a/openspec/changes/export-import-prompts-and-assistants/design.md b/openspec/changes/export-import-prompts-and-assistants/design.md index 22dcf55bb..e2d50b60f 100644 --- a/openspec/changes/export-import-prompts-and-assistants/design.md +++ b/openspec/changes/export-import-prompts-and-assistants/design.md @@ -121,6 +121,7 @@ A mise task runs a short Python script that pulls the imported prompt with `incl - [Pushing a manifest identical to the latest commit may return 409] → Import compares first and pushes nothing when the prompt matches. - [langstar finds a deployment's Agent Server URL only in `source_config.custom_url` (`resolve_deployment_url` in `cli/src/deployment_utils.rs`), and a deployment built from GitHub has none, so every `langstar assistant` command, the new ones included, fails on such a deployment] → In [export-import-prompts-and-assistants Task 2.2 Return the URL of a deployment built from GitHub](tasks.md#L13), a developer adds the control plane's top-level `url` to the SDK's `Deployment`, which drops it today, and makes `resolve_deployment_url` use it, before anyone builds the assistant export, compare, verify and import commands. - [The SDK changes for export and import add public fields to `Deployment`, `Prompt`, `CommitManifestResponse`, `CommitData`, `CommitRequest`, `Assistant` and the assistant request structs, so a crate that builds one of those structs with a struct literal or destructures it with a pattern that names every field stops compiling] → Each pull request that adds a field updates every struct literal of that struct in this repository. [Should the SDK field additions count as a breaking change?](#9---should-the-sdk-field-additions-count-as-a-breaking-change) asks codekiln whether those pull requests also carry a `BREAKING CHANGE:` footer, which `docs/dev/git-scm-conventions.md` says makes the release a major version. +- [A prompt's manifest may hold a credential written as a literal model setting, such as an API key in a header, rather than as a secret reference] → The secret reference walk finds only `{"lc": 1, "type": "secret"}` objects, and langstar cannot tell a credential from any other string in a manifest, so export writes the manifest as LangSmith returns it. The prompt export docs in [export-import-prompts-and-assistants Task 5.2 Document prompt export](tasks.md#L52) tell users to keep credentials in the workspace's secrets and to read an export file before committing it. - [An assistant's configuration, context or metadata may hold a credential, which export copies into a file meant for Git] → Langstar cannot tell a credential from any other string in an assistant's configuration, as it cannot tell which key holds a prompt name, so export writes those objects as the Agent Server returns them. The assistant export docs in [export-import-prompts-and-assistants Task 8.2 Document assistant export](tasks.md#L104) tell users to keep credentials in the deployment's environment variables or the workspace's secrets rather than in an assistant, and to read an export file before committing it. - [A tag in the destination may already point at another commit] → `--update` moves the tag, and `--dry-run` lists the move. - [`--update` deletes a tag on the matching destination commit that the file lacks, and a graph on a deployment may read that tag] → `--dry-run` lists each delete, and `assistant verify` reports a prompt tag an assistant refers to that the workspace no longer has. diff --git a/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md index de6283e8b..0216913a5 100644 --- a/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md +++ b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md @@ -52,7 +52,7 @@ Exporting the same commit twice SHALL produce identical files when nobody change ### Requirement: List the credentials a prompt needs without their values -The export file SHALL list, under `requires.secrets`, the name of every secret the manifest refers to, and under `requires.model_class`, the class name that ends the chat model constructor's `id`, such as `ChatOpenAI`. It SHALL hold no secret values. Import and verify SHALL read the secret names and the model class out of the file's manifest again, and when either differs from `requires.secrets` or `requires.model_class`, write nothing, name the field and exit with status 1. +The export file SHALL list, under `requires.secrets`, the name of every secret the manifest refers to, and under `requires.model_class`, the class name that ends the chat model constructor's `id`, such as `ChatOpenAI`. It SHALL hold no value of a secret the manifest refers to. Import and verify SHALL read both again from the file's manifest, and when either differs from `requires.secrets` or `requires.model_class`, write nothing, name the field and exit with status 1. #### Scenario: Model reads an API key secret diff --git a/openspec/changes/export-import-prompts-and-assistants/tasks.md b/openspec/changes/export-import-prompts-and-assistants/tasks.md index 97af913cf..c15e0efc1 100644 --- a/openspec/changes/export-import-prompts-and-assistants/tasks.md +++ b/openspec/changes/export-import-prompts-and-assistants/tasks.md @@ -49,7 +49,7 @@ Each task in this group adds a public field to one of these SDK structs: `Prompt - An error test that exports a handle the workspace has no prompt for, and asserts that it exits with status 1, names the handle and writes no file. - A CLI test that runs `prompt export` without `--output`, parses all of stdout as one JSON document, and asserts that its `source.commit_hash` matches the SDK's read and that any progress message is on stderr. - An integration test that gives one commit two tags, exports it twice, and asserts that the two files are identical. -- [ ] 5.2 Document `prompt export` in `docs/usage/` and the README; verify that each documented command runs as written +- [ ] 5.2 Document `prompt export` in `docs/usage/` and the README. The page says that export writes the manifest as LangSmith returns it, so a credential written into a model setting as a literal value, rather than as a secret reference, reaches the file; tells users to keep credentials in the workspace's secrets; and tells them to read an export file before they commit it. Verify that each documented command runs as written. ## 6. prompt compare and verify From 432aa10ab3c41a832228f003f598c116df1c3dcb Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Wed, 14 Oct 2026 00:42:44 -0400 Subject: [PATCH 45/90] =?UTF-8?q?=F0=9F=93=9A=20docs(openspec):=20refuse?= =?UTF-8?q?=20to=20export=20a=20prompt=20whose=20model=20holds=20a=20liter?= =?UTF-8?q?al=20API=20key?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit #807 Export, import, compare, and verify prompts and assistants across LangSmith environments Co-Authored-By: Claude Opus 5.5 --- .../export-import-prompts-and-assistants/design.md | 2 +- .../specs/prompt/spec.md | 10 ++++++++++ .../export-import-prompts-and-assistants/tasks.md | 2 +- 3 files changed, 12 insertions(+), 2 deletions(-) diff --git a/openspec/changes/export-import-prompts-and-assistants/design.md b/openspec/changes/export-import-prompts-and-assistants/design.md index e2d50b60f..0cf64635b 100644 --- a/openspec/changes/export-import-prompts-and-assistants/design.md +++ b/openspec/changes/export-import-prompts-and-assistants/design.md @@ -121,7 +121,7 @@ A mise task runs a short Python script that pulls the imported prompt with `incl - [Pushing a manifest identical to the latest commit may return 409] → Import compares first and pushes nothing when the prompt matches. - [langstar finds a deployment's Agent Server URL only in `source_config.custom_url` (`resolve_deployment_url` in `cli/src/deployment_utils.rs`), and a deployment built from GitHub has none, so every `langstar assistant` command, the new ones included, fails on such a deployment] → In [export-import-prompts-and-assistants Task 2.2 Return the URL of a deployment built from GitHub](tasks.md#L13), a developer adds the control plane's top-level `url` to the SDK's `Deployment`, which drops it today, and makes `resolve_deployment_url` use it, before anyone builds the assistant export, compare, verify and import commands. - [The SDK changes for export and import add public fields to `Deployment`, `Prompt`, `CommitManifestResponse`, `CommitData`, `CommitRequest`, `Assistant` and the assistant request structs, so a crate that builds one of those structs with a struct literal or destructures it with a pattern that names every field stops compiling] → Each pull request that adds a field updates every struct literal of that struct in this repository. [Should the SDK field additions count as a breaking change?](#9---should-the-sdk-field-additions-count-as-a-breaking-change) asks codekiln whether those pull requests also carry a `BREAKING CHANGE:` footer, which `docs/dev/git-scm-conventions.md` says makes the release a major version. -- [A prompt's manifest may hold a credential written as a literal model setting, such as an API key in a header, rather than as a secret reference] → The secret reference walk finds only `{"lc": 1, "type": "secret"}` objects, and langstar cannot tell a credential from any other string in a manifest, so export writes the manifest as LangSmith returns it. The prompt export docs in [export-import-prompts-and-assistants Task 5.2 Document prompt export](tasks.md#L52) tell users to keep credentials in the workspace's secrets and to read an export file before committing it. +- [A prompt's manifest may hold a credential written as a literal model setting, rather than as a secret reference] → Export refuses with status 1 and writes nothing when a chat model setting whose name ends in `api_key` holds a string, and names that setting's path without its value. A credential under any other name, such as a header, still reaches the file, because langstar cannot tell it from any other string. The prompt export docs in [export-import-prompts-and-assistants Task 5.2 Document prompt export](tasks.md#L52) tell users to keep credentials in the workspace's secrets and to read an export file before committing it. - [An assistant's configuration, context or metadata may hold a credential, which export copies into a file meant for Git] → Langstar cannot tell a credential from any other string in an assistant's configuration, as it cannot tell which key holds a prompt name, so export writes those objects as the Agent Server returns them. The assistant export docs in [export-import-prompts-and-assistants Task 8.2 Document assistant export](tasks.md#L104) tell users to keep credentials in the deployment's environment variables or the workspace's secrets rather than in an assistant, and to read an export file before committing it. - [A tag in the destination may already point at another commit] → `--update` moves the tag, and `--dry-run` lists the move. - [`--update` deletes a tag on the matching destination commit that the file lacks, and a graph on a deployment may read that tag] → `--dry-run` lists each delete, and `assistant verify` reports a prompt tag an assistant refers to that the workspace no longer has. diff --git a/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md index 0216913a5..cd4121670 100644 --- a/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md +++ b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md @@ -66,6 +66,16 @@ The export file SHALL list, under `requires.secrets`, the name of every secret t - **AND** a user runs `langstar prompt import my-prompt.json --workspace-id ` - **THEN** langstar creates no prompt, names `requires.secrets` and exits with status 1 +### Requirement: Refuse to export an API key written into a prompt's model + +When a chat model constructor in the manifest holds a string, rather than a secret reference, under a setting whose name ends in `api_key`, such as `openai_api_key`, `langstar prompt export` SHALL write no file and nothing to stdout, name that setting's JSON Pointer path without its value, and exit with status 1. + +#### Scenario: Model holds a literal API key + +- **WHEN** the prompt's `ChatOpenAI` constructor holds a string under `openai_api_key` +- **AND** a user runs `langstar prompt export my-prompt --output my-prompt.json` +- **THEN** langstar writes no file, names the path of `openai_api_key` without its value and exits with status 1 + ### Requirement: Export, import and verify a prompt without a model When the manifest holds no chat model constructor, `langstar prompt export` SHALL write `requires.model_class` as null. Import and verify SHALL accept null when the manifest they read again holds no chat model, skip the saved model configuration check, report no note for it, and still run the secret check. diff --git a/openspec/changes/export-import-prompts-and-assistants/tasks.md b/openspec/changes/export-import-prompts-and-assistants/tasks.md index c15e0efc1..7802806f0 100644 --- a/openspec/changes/export-import-prompts-and-assistants/tasks.md +++ b/openspec/changes/export-import-prompts-and-assistants/tasks.md @@ -39,7 +39,7 @@ Each task in this group adds a public field to one of these SDK structs: `Prompt - A test that a changed `id` array inside `commit.manifest`, such as `ChatOpenAI` replaced by `ChatAnthropic`, is reported at its path. - A test that a destination whose `commit.model_config` and `commit.model_provider` differ from the file's, while the manifest matches, is reported as a match. - [ ] 4.3 Add the shared result report and the exit statuses 0, 3 and 1, printing JSON to stdout and progress to stderr, and leave clap's status 2 for usage errors; verify with unit tests of the report and of the exit status each result gives -- [ ] 4.4 Add the secret reference walk that lists secret names from a manifest; verify with unit tests on a copy of the `prompt_with_model` manifest from the LangSmith Python SDK's integration tests, which `references/langsmith-sdk.md` links, and on a manifest with no secret reference +- [ ] 4.4 Add the secret reference walk that lists secret names from a manifest and returns the JSON Pointer path of each chat model setting whose name ends in `api_key` and holds a string instead of a secret reference, which `prompt export` from 5.1 refuses with status 1; verify with unit tests on a copy of the `prompt_with_model` manifest from the LangSmith Python SDK's integration tests, which `references/langsmith-sdk.md` links, on a manifest with no secret reference, and on a manifest whose `openai_api_key` holds a string, asserting that the walk returns that setting's path and that the error message holds the path and not the string ## 5. prompt export From 544ff3cf5a0cc3b6872ebb9b57b6172088b26ddc Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Wed, 14 Oct 2026 00:51:43 -0400 Subject: [PATCH 46/90] =?UTF-8?q?=F0=9F=93=9A=20docs(openspec):=20say=20ex?= =?UTF-8?q?port=20refuses=20a=20literal=20API=20key=20in=20task=205.2,=20t?= =?UTF-8?q?est=20that=20refusal,=20and=20narrow=20what=20an=20interrupted?= =?UTF-8?q?=20import=20finishes?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit #807 Export, import, compare, and verify prompts and assistants across LangSmith environments Co-Authored-By: Claude Opus 5.5 --- .../changes/export-import-prompts-and-assistants/design.md | 6 +++--- .../export-import-prompts-and-assistants/proposal.md | 2 +- .../specs/prompt/spec.md | 2 +- .../changes/export-import-prompts-and-assistants/tasks.md | 3 ++- 4 files changed, 7 insertions(+), 6 deletions(-) diff --git a/openspec/changes/export-import-prompts-and-assistants/design.md b/openspec/changes/export-import-prompts-and-assistants/design.md index 0cf64635b..622aec448 100644 --- a/openspec/changes/export-import-prompts-and-assistants/design.md +++ b/openspec/changes/export-import-prompts-and-assistants/design.md @@ -63,7 +63,7 @@ Export reads the commit with `include=model`, the form the Go SDK uses (`referen Import writes the IDs of what it created to `.destinations.json`, keyed by destination workspace ID for prompts and by deployment ID for assistants. The export file stays the same after any number of imports, and a fresh export never wipes out the record of earlier imports. Import replaces the record by writing the whole new record to a temporary file in the same directory and renaming it over the old one, so a langstar that stops in the middle of a write leaves the earlier record whole, pending assistant IDs included. -A repeated prompt import finds the copy by its handle and uses the record to tell whether someone changed it. Import writes the new commit hash to the record before it reads the commit back, so a read-back that finds a difference still leaves the hash in the record. The next import compares the destination's latest commit with that hash, and reports a change only when someone else pushed a commit after langstar's. An import that stopped after its push and before its record write leaves the earlier hash, or no entry, in the record, so when the destination's latest commit matches the file, the next import writes that commit's hash to the record and reports no change. +A repeated prompt import finds the copy by its handle and uses the record to tell whether someone changed it. Import writes the new commit hash to the record before it reads the commit back, so a read-back that finds a difference still leaves the hash in the record. The next import compares the destination's latest commit with that hash, and reports a change only when someone else pushed a commit after langstar's. Langstar can stop after it pushes a commit and before it writes that commit's hash to the record. The next import finds the destination's latest commit matching the file, writes its hash to the record and reports the prompt unchanged. An assistant's ID differs on every deployment, so the record is the only way a repeated import finds the assistant it made. Import chooses the new assistant's ID itself and writes it to the record, marked pending, before it sends the create request with that `assistant_id` and `if_exists` set to `do_nothing`, which the Agent Server's `AssistantCreate` accepts. After the create succeeds, import marks the entry created. If langstar stops before then, the next import finds the pending entry and sends the same create again, and the Agent Server returns the assistant the first import made. @@ -121,8 +121,8 @@ A mise task runs a short Python script that pulls the imported prompt with `incl - [Pushing a manifest identical to the latest commit may return 409] → Import compares first and pushes nothing when the prompt matches. - [langstar finds a deployment's Agent Server URL only in `source_config.custom_url` (`resolve_deployment_url` in `cli/src/deployment_utils.rs`), and a deployment built from GitHub has none, so every `langstar assistant` command, the new ones included, fails on such a deployment] → In [export-import-prompts-and-assistants Task 2.2 Return the URL of a deployment built from GitHub](tasks.md#L13), a developer adds the control plane's top-level `url` to the SDK's `Deployment`, which drops it today, and makes `resolve_deployment_url` use it, before anyone builds the assistant export, compare, verify and import commands. - [The SDK changes for export and import add public fields to `Deployment`, `Prompt`, `CommitManifestResponse`, `CommitData`, `CommitRequest`, `Assistant` and the assistant request structs, so a crate that builds one of those structs with a struct literal or destructures it with a pattern that names every field stops compiling] → Each pull request that adds a field updates every struct literal of that struct in this repository. [Should the SDK field additions count as a breaking change?](#9---should-the-sdk-field-additions-count-as-a-breaking-change) asks codekiln whether those pull requests also carry a `BREAKING CHANGE:` footer, which `docs/dev/git-scm-conventions.md` says makes the release a major version. -- [A prompt's manifest may hold a credential written as a literal model setting, rather than as a secret reference] → Export refuses with status 1 and writes nothing when a chat model setting whose name ends in `api_key` holds a string, and names that setting's path without its value. A credential under any other name, such as a header, still reaches the file, because langstar cannot tell it from any other string. The prompt export docs in [export-import-prompts-and-assistants Task 5.2 Document prompt export](tasks.md#L52) tell users to keep credentials in the workspace's secrets and to read an export file before committing it. -- [An assistant's configuration, context or metadata may hold a credential, which export copies into a file meant for Git] → Langstar cannot tell a credential from any other string in an assistant's configuration, as it cannot tell which key holds a prompt name, so export writes those objects as the Agent Server returns them. The assistant export docs in [export-import-prompts-and-assistants Task 8.2 Document assistant export](tasks.md#L104) tell users to keep credentials in the deployment's environment variables or the workspace's secrets rather than in an assistant, and to read an export file before committing it. +- [A prompt's manifest may hold a credential written as a literal model setting, rather than as a secret reference] → Export refuses with status 1 and writes nothing when a chat model setting whose name ends in `api_key` holds a string, and names that setting's path without its value. A credential under any other name, such as a header, still reaches the file, because langstar cannot tell it from any other string. The prompt export docs in [export-import-prompts-and-assistants Task 5.2 Document prompt export](tasks.md#L53) tell users to keep credentials in the workspace's secrets and to read an export file before committing it. +- [An assistant's configuration, context or metadata may hold a credential, which export copies into a file meant for Git] → Langstar cannot tell a credential from any other string in an assistant's configuration, as it cannot tell which key holds a prompt name, so export writes those objects as the Agent Server returns them. The assistant export docs in [export-import-prompts-and-assistants Task 8.2 Document assistant export](tasks.md#L105) tell users to keep credentials in the deployment's environment variables or the workspace's secrets rather than in an assistant, and to read an export file before committing it. - [A tag in the destination may already point at another commit] → `--update` moves the tag, and `--dry-run` lists the move. - [`--update` deletes a tag on the matching destination commit that the file lacks, and a graph on a deployment may read that tag] → `--dry-run` lists each delete, and `assistant verify` reports a prompt tag an assistant refers to that the workspace no longer has. - [The control plane may refuse a second deployment from this repository with 409] → In [export-import-prompts-and-assistants Task 2.3 Register the test graph under a second graph ID](tasks.md#L18), a developer deploys the new `langgraph.json` once, before anyone builds the fixture. If the control plane answers with 409, the developer brings [Which second workspace and second deployment should the tests use?](#5---which-second-workspace-and-second-deployment-should-the-tests-use) back to codekiln. diff --git a/openspec/changes/export-import-prompts-and-assistants/proposal.md b/openspec/changes/export-import-prompts-and-assistants/proposal.md index 5b59b07c4..1480dc6fe 100644 --- a/openspec/changes/export-import-prompts-and-assistants/proposal.md +++ b/openspec/changes/export-import-prompts-and-assistants/proposal.md @@ -11,7 +11,7 @@ A prompt belongs to a workspace and an assistant belongs to a deployment, but bo ## What Changes - `langstar prompt export ` writes one JSON file holding a saved prompt commit as LangSmith returns it with the model included: message templates, input variables, output schema, structured output method, model binding and commit description, plus the repo's description, readme, tags and visibility, the commit tags, and where it came from. -- `langstar prompt import --workspace-id ` creates the prompt in the destination workspace, or adds a commit to an existing prompt, updates its repo details and creates, moves or deletes its commit tags when the user passes `--update`. An import that stopped after creating the repo finishes on the next run. `--dry-run` shows what it would do. After writing, it reads the prompt back and reports any difference. +- `langstar prompt import --workspace-id ` creates the prompt in the destination workspace, or adds a commit to an existing prompt, updates its repo details and creates, moves or deletes its commit tags when the user passes `--update`. An import that stopped before it pushed the first commit finishes on the next run, and one that stopped before it created the commit tags finishes with `--update`. `--dry-run` shows what it would do. After writing, it reads the prompt back and reports any difference. - `langstar prompt compare ` reports the differences between the file and a prompt in a workspace. `langstar prompt verify ` also checks that the workspace has the secrets the prompt's model reads, and notes when the workspace has no saved model configuration for the model's provider. - `langstar assistant export`, `import`, `compare` and `verify` do the same for assistants on a deployment. The file holds the name, description, graph ID, configuration, context, metadata and any prompt names the assistant refers to. `verify` checks that the destination deployment has the graph and that the destination workspace has the referenced prompts. - Import records each destination ID (the new commit hash, or the new assistant ID) in a separate file next to the export. When a user imports the same file again, langstar reads that file to find the assistant it created earlier instead of creating another one, and to tell whether someone pushed a commit to the prompt since the last import. diff --git a/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md index cd4121670..334df62e8 100644 --- a/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md +++ b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md @@ -215,7 +215,7 @@ When the destination record holds a commit for the destination workspace and the ### Requirement: Record a matching commit that an earlier import pushed -When the destination's latest commit matches the file and the destination record holds no commit for the destination workspace or a different one, `langstar prompt import` without `--dry-run` SHALL write that commit's hash to the record, whatever hash the record held, and SHALL NOT say that someone changed the prompt. When the commit tags match the file too, it SHALL report the prompt unchanged and exit with status 0. +When the destination's latest commit matches the file, `langstar prompt import` without `--dry-run` SHALL write that commit's hash to the destination record under the destination workspace. When the commit tags match the file too, it SHALL report the prompt unchanged and exit with status 0. #### Scenario: Earlier import stopped before it wrote the record diff --git a/openspec/changes/export-import-prompts-and-assistants/tasks.md b/openspec/changes/export-import-prompts-and-assistants/tasks.md index 7802806f0..0f7cc09d1 100644 --- a/openspec/changes/export-import-prompts-and-assistants/tasks.md +++ b/openspec/changes/export-import-prompts-and-assistants/tasks.md @@ -49,7 +49,8 @@ Each task in this group adds a public field to one of these SDK structs: `Prompt - An error test that exports a handle the workspace has no prompt for, and asserts that it exits with status 1, names the handle and writes no file. - A CLI test that runs `prompt export` without `--output`, parses all of stdout as one JSON document, and asserts that its `source.commit_hash` matches the SDK's read and that any progress message is on stderr. - An integration test that gives one commit two tags, exports it twice, and asserts that the two files are identical. -- [ ] 5.2 Document `prompt export` in `docs/usage/` and the README. The page says that export writes the manifest as LangSmith returns it, so a credential written into a model setting as a literal value, rather than as a secret reference, reaches the file; tells users to keep credentials in the workspace's secrets; and tells them to read an export file before they commit it. Verify that each documented command runs as written. + - A CLI test that serves, through httpmock, a commit whose `ChatOpenAI` constructor holds a string under `openai_api_key`, runs `prompt export my-prompt --output my-prompt.json`, and asserts that it exits with status 1, writes no file and nothing to stdout, and names the setting's path on stderr without the string. +- [ ] 5.2 Document `prompt export` in `docs/usage/` and the README. The page says that export refuses a prompt whose model holds an API key as a literal string, names that setting and writes no file, and that a credential kept under any other name, such as a header, still reaches the file; tells users to keep credentials in the workspace's secrets; and tells them to read an export file before they commit it. Verify that each documented command runs as written. ## 6. prompt compare and verify From 7be9831347c8793f2a571dcf3b94b460d7976eec Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Wed, 14 Oct 2026 01:06:21 -0400 Subject: [PATCH 47/90] =?UTF-8?q?=F0=9F=93=9A=20docs(openspec):=20untangle?= =?UTF-8?q?=20the=20destination-changed=20warning,=20match=20saved=20model?= =?UTF-8?q?=20configurations=20on=20the=20model=20class,=20and=20fill=20so?= =?UTF-8?q?urce.workspace=5Fid=20from=20the=20repo's=20tenant=5Fid?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit #807 Export, import, compare, and verify prompts and assistants across LangSmith environments Co-Authored-By: Claude Opus 5.5 --- .../changes/export-import-prompts-and-assistants/design.md | 4 ++-- .../changes/export-import-prompts-and-assistants/proposal.md | 2 +- .../export-import-prompts-and-assistants/specs/prompt/spec.md | 4 ++-- .../changes/export-import-prompts-and-assistants/tasks.md | 3 ++- 4 files changed, 7 insertions(+), 6 deletions(-) diff --git a/openspec/changes/export-import-prompts-and-assistants/design.md b/openspec/changes/export-import-prompts-and-assistants/design.md index 622aec448..ca6047842 100644 --- a/openspec/changes/export-import-prompts-and-assistants/design.md +++ b/openspec/changes/export-import-prompts-and-assistants/design.md @@ -121,8 +121,8 @@ A mise task runs a short Python script that pulls the imported prompt with `incl - [Pushing a manifest identical to the latest commit may return 409] → Import compares first and pushes nothing when the prompt matches. - [langstar finds a deployment's Agent Server URL only in `source_config.custom_url` (`resolve_deployment_url` in `cli/src/deployment_utils.rs`), and a deployment built from GitHub has none, so every `langstar assistant` command, the new ones included, fails on such a deployment] → In [export-import-prompts-and-assistants Task 2.2 Return the URL of a deployment built from GitHub](tasks.md#L13), a developer adds the control plane's top-level `url` to the SDK's `Deployment`, which drops it today, and makes `resolve_deployment_url` use it, before anyone builds the assistant export, compare, verify and import commands. - [The SDK changes for export and import add public fields to `Deployment`, `Prompt`, `CommitManifestResponse`, `CommitData`, `CommitRequest`, `Assistant` and the assistant request structs, so a crate that builds one of those structs with a struct literal or destructures it with a pattern that names every field stops compiling] → Each pull request that adds a field updates every struct literal of that struct in this repository. [Should the SDK field additions count as a breaking change?](#9---should-the-sdk-field-additions-count-as-a-breaking-change) asks codekiln whether those pull requests also carry a `BREAKING CHANGE:` footer, which `docs/dev/git-scm-conventions.md` says makes the release a major version. -- [A prompt's manifest may hold a credential written as a literal model setting, rather than as a secret reference] → Export refuses with status 1 and writes nothing when a chat model setting whose name ends in `api_key` holds a string, and names that setting's path without its value. A credential under any other name, such as a header, still reaches the file, because langstar cannot tell it from any other string. The prompt export docs in [export-import-prompts-and-assistants Task 5.2 Document prompt export](tasks.md#L53) tell users to keep credentials in the workspace's secrets and to read an export file before committing it. -- [An assistant's configuration, context or metadata may hold a credential, which export copies into a file meant for Git] → Langstar cannot tell a credential from any other string in an assistant's configuration, as it cannot tell which key holds a prompt name, so export writes those objects as the Agent Server returns them. The assistant export docs in [export-import-prompts-and-assistants Task 8.2 Document assistant export](tasks.md#L105) tell users to keep credentials in the deployment's environment variables or the workspace's secrets rather than in an assistant, and to read an export file before committing it. +- [A prompt's manifest may hold a credential written as a literal model setting, rather than as a secret reference] → Export refuses with status 1 and writes nothing when a chat model setting whose name ends in `api_key` holds a string, and names that setting's path without its value. A credential under any other name, such as a header, still reaches the file, because langstar cannot tell it from any other string. The prompt export docs in [export-import-prompts-and-assistants Task 5.2 Document prompt export](tasks.md#L54) tell users to keep credentials in the workspace's secrets and to read an export file before committing it. +- [An assistant's configuration, context or metadata may hold a credential, which export copies into a file meant for Git] → Langstar cannot tell a credential from any other string in an assistant's configuration, as it cannot tell which key holds a prompt name, so export writes those objects as the Agent Server returns them. The assistant export docs in [export-import-prompts-and-assistants Task 8.2 Document assistant export](tasks.md#L106) tell users to keep credentials in the deployment's environment variables or the workspace's secrets rather than in an assistant, and to read an export file before committing it. - [A tag in the destination may already point at another commit] → `--update` moves the tag, and `--dry-run` lists the move. - [`--update` deletes a tag on the matching destination commit that the file lacks, and a graph on a deployment may read that tag] → `--dry-run` lists each delete, and `assistant verify` reports a prompt tag an assistant refers to that the workspace no longer has. - [The control plane may refuse a second deployment from this repository with 409] → In [export-import-prompts-and-assistants Task 2.3 Register the test graph under a second graph ID](tasks.md#L18), a developer deploys the new `langgraph.json` once, before anyone builds the fixture. If the control plane answers with 409, the developer brings [Which second workspace and second deployment should the tests use?](#5---which-second-workspace-and-second-deployment-should-the-tests-use) back to codekiln. diff --git a/openspec/changes/export-import-prompts-and-assistants/proposal.md b/openspec/changes/export-import-prompts-and-assistants/proposal.md index 1480dc6fe..c6b38259a 100644 --- a/openspec/changes/export-import-prompts-and-assistants/proposal.md +++ b/openspec/changes/export-import-prompts-and-assistants/proposal.md @@ -12,7 +12,7 @@ A prompt belongs to a workspace and an assistant belongs to a deployment, but bo - `langstar prompt export ` writes one JSON file holding a saved prompt commit as LangSmith returns it with the model included: message templates, input variables, output schema, structured output method, model binding and commit description, plus the repo's description, readme, tags and visibility, the commit tags, and where it came from. - `langstar prompt import --workspace-id ` creates the prompt in the destination workspace, or adds a commit to an existing prompt, updates its repo details and creates, moves or deletes its commit tags when the user passes `--update`. An import that stopped before it pushed the first commit finishes on the next run, and one that stopped before it created the commit tags finishes with `--update`. `--dry-run` shows what it would do. After writing, it reads the prompt back and reports any difference. -- `langstar prompt compare ` reports the differences between the file and a prompt in a workspace. `langstar prompt verify ` also checks that the workspace has the secrets the prompt's model reads, and notes when the workspace has no saved model configuration for the model's provider. +- `langstar prompt compare ` reports the differences between the file and a prompt in a workspace. `langstar prompt verify ` also checks that the workspace has the secrets the prompt's model reads, and notes when the workspace has no saved model configuration for the model's class, such as `ChatOpenAI`. - `langstar assistant export`, `import`, `compare` and `verify` do the same for assistants on a deployment. The file holds the name, description, graph ID, configuration, context, metadata and any prompt names the assistant refers to. `verify` checks that the destination deployment has the graph and that the destination workspace has the referenced prompts. - Import records each destination ID (the new commit hash, or the new assistant ID) in a separate file next to the export. When a user imports the same file again, langstar reads that file to find the assistant it created earlier instead of creating another one, and to tell whether someone pushed a commit to the prompt since the last import. - Import runs the dependency checks of `verify` before it writes, and writes nothing when one fails. diff --git a/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md index 334df62e8..c2409c645 100644 --- a/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md +++ b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md @@ -205,7 +205,7 @@ After import writes a commit, langstar SHALL record the destination workspace, h ### Requirement: Warn when the destination changed since the last import -When the destination record holds a commit for the destination workspace and the destination's latest commit is a different one that also differs from the file, `langstar prompt import` SHALL say that someone changed the prompt after the last import, in addition to listing the differences. +When the destination record holds a commit for the destination workspace, and the destination's latest commit is not that commit and differs from the file, `langstar prompt import` SHALL say that someone changed the prompt after the last import, and list the differences. #### Scenario: Someone edited the destination prompt @@ -294,7 +294,7 @@ After writing, `langstar prompt import` SHALL record the commit hash in the dest - **AND** the file lists `OPENAI_API_KEY` under `requires.secrets` - **THEN** langstar reports `OPENAI_API_KEY` as missing and exits with status 3 -#### Scenario: No saved model configuration for the provider +#### Scenario: No saved model configuration for the model class - **WHEN** the file's `requires.model_class` is `ChatOpenAI` - **AND** no saved model configuration in the destination workspace has a settings `id` ending in `ChatOpenAI` diff --git a/openspec/changes/export-import-prompts-and-assistants/tasks.md b/openspec/changes/export-import-prompts-and-assistants/tasks.md index 0f7cc09d1..5c939cb4f 100644 --- a/openspec/changes/export-import-prompts-and-assistants/tasks.md +++ b/openspec/changes/export-import-prompts-and-assistants/tasks.md @@ -25,7 +25,7 @@ Each task in this group adds a public field to one of these SDK structs: `Prompt - [ ] 3.1 Add `description`, `model_config` and `model_provider` to `CommitManifestResponse`, and a commit read that sends `include=model` and returns that struct, to `sdk/src/prompts.rs`; verify with an httpmock test that asserts the query string and the parsed fields, and an integration test that reads a prompt with a model - [ ] 3.2 Add the commit's `id` to `CommitData` in `sdk/src/prompts.rs`, a call that lists a prompt's commits with their `id` and hash (`GET /api/v1/commits/{owner}/{repo}`), calls to list, create and move commit tags, which name their commit by that `id` in `POST /api/v1/repos/{owner}/{repo}/tags` and `PATCH /api/v1/repos/{owner}/{repo}/tags/{tag_name}`, and a call that deletes a tag with `DELETE /api/v1/repos/{owner}/{repo}/tags/{tag_name}`, which `prompt import --update` from 7.2 and the end-to-end test in 10.3 use; verify with httpmock tests that assert `commit_id` in each create and move request body and the path of the delete request, and an integration test that pushes two commits, tags the first, finds the second commit's `id` through the commit list, moves the tag to it, reads the tag back through the SDK, deletes it, and asserts that the tag list no longer holds it - [ ] 3.3 Add `context` and `version` to `Assistant`, `context` and `graph_id` to `UpdateAssistantRequest`, `context`, `assistant_id` and `if_exists` to `CreateAssistantRequest`, and a `graph_id` filter to `AssistantSearchRequest`, in `sdk/src/assistants.rs`; verify with httpmock tests of the request bodies and an integration test that creates an assistant with context on the destination deployment from 2.4, reads it back with its version, finds it by graph ID, moves it from `test_graph` to `test_graph_alt`, and reads `test_graph_alt` back. The test runs on the destination deployment because the source deployment serves only `test_graph`, and task 2.4, which adds the destination fixture, merges before this group. -- [ ] 3.4 Add `description` to `CommitRequest`, `readme` and the repo's `tags` to `Prompt`, which `PromptClient::get` returns and export reads the repo details from, and a repo update that sends `PATCH /api/v1/repos/{owner}/{repo}` with `description`, `readme`, `tags` and `is_public`, to `sdk/src/prompts.rs`; verify with httpmock tests of the request bodies and of a `PromptClient::get` response that holds `readme` and `tags`, and an integration test that pushes a commit with a description and reads it back through the commit read from 3.1, then updates the repo's description, readme, tags and visibility and reads each back through `PromptClient::get` +- [ ] 3.4 Add `description` to `CommitRequest`, and `readme`, the repo's `tags` and `tenant_id` to `Prompt`, which `PromptClient::get` returns and export reads the repo details and `source.workspace_id` from, and a repo update that sends `PATCH /api/v1/repos/{owner}/{repo}` with `description`, `readme`, `tags` and `is_public`, to `sdk/src/prompts.rs`; verify with httpmock tests of the request bodies and of a `PromptClient::get` response that holds `readme`, `tags` and `tenant_id`, and an integration test that pushes a commit with a description and reads it back through the commit read from 3.1, then updates the repo's description, readme, tags and visibility and reads each back through `PromptClient::get` ## 4. Shared code for the prompt and assistant commands: the export file, the comparison and the report @@ -50,6 +50,7 @@ Each task in this group adds a public field to one of these SDK structs: `Prompt - A CLI test that runs `prompt export` without `--output`, parses all of stdout as one JSON document, and asserts that its `source.commit_hash` matches the SDK's read and that any progress message is on stderr. - An integration test that gives one commit two tags, exports it twice, and asserts that the two files are identical. - A CLI test that serves, through httpmock, a commit whose `ChatOpenAI` constructor holds a string under `openai_api_key`, runs `prompt export my-prompt --output my-prompt.json`, and asserts that it exits with status 1, writes no file and nothing to stdout, and names the setting's path on stderr without the string. + - A CLI test that serves a prompt through httpmock, runs `prompt export my-prompt` with no `--workspace-id` and no `LANGSMITH_WORKSPACE_ID`, and asserts that `source.workspace_id` holds the repo's `tenant_id`. - [ ] 5.2 Document `prompt export` in `docs/usage/` and the README. The page says that export refuses a prompt whose model holds an API key as a literal string, names that setting and writes no file, and that a credential kept under any other name, such as a header, still reaches the file; tells users to keep credentials in the workspace's secrets; and tells them to read an export file before they commit it. Verify that each documented command runs as written. ## 6. prompt compare and verify From bd07891b868a34b1acb116c7de9b05668a1c1580 Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Wed, 14 Oct 2026 01:26:23 -0400 Subject: [PATCH 48/90] =?UTF-8?q?=F0=9F=93=9A=20docs(openspec):=20give=20t?= =?UTF-8?q?he=20deployment=20url=20fix=20a=20Decision,=20fold=20the=20mode?= =?UTF-8?q?l=5Fconfig=20paragraph=20into=20compare,=20untangle=20the=20ass?= =?UTF-8?q?istant=20credential=20mitigation,=20and=20list=20the=20fields?= =?UTF-8?q?=20upstream=20update=20patches?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit #807 Export, import, compare, and verify prompts and assistants across LangSmith environments Co-Authored-By: Claude Opus 5.5 --- .../export-import-prompts-and-assistants/design.md | 13 +++++++------ .../references/langgraph-sdk.md | 2 +- 2 files changed, 8 insertions(+), 7 deletions(-) diff --git a/openspec/changes/export-import-prompts-and-assistants/design.md b/openspec/changes/export-import-prompts-and-assistants/design.md index ca6047842..c1f8f0eac 100644 --- a/openspec/changes/export-import-prompts-and-assistants/design.md +++ b/openspec/changes/export-import-prompts-and-assistants/design.md @@ -4,7 +4,7 @@ `langstar prompt push` in `cli/src/commands/prompt.rs` builds one shape of prompt: a single human message, with an optional schema from [#402 structured output prompts](https://github.com/codekiln/langstar/issues/402). `langstar prompt pull` prints any manifest LangSmith returns, but `PromptClient` in `sdk/src/prompts.rs` never asks LangSmith for the model, so that manifest has none. `PromptClient` keeps the manifest as raw JSON and has a typed `StructuredPrompt` view that holds messages, the schema and the method. -`langstar assistant` in `cli/src/commands/assistant.rs` takes `--deployment` on every subcommand and sends requests to that deployment's Agent Server. `AssistantClient` in `sdk/src/assistants.rs` has no `context` or `version` field and its search cannot filter by graph ID. +`langstar assistant` in `cli/src/commands/assistant.rs` takes `--deployment` on every subcommand and sends its requests to the Agent Server URL that `resolve_deployment_url` in `cli/src/deployment_utils.rs` reads from the deployment's `source_config.custom_url`. A deployment built from GitHub has no `custom_url`, so every `langstar assistant` command fails on one today. `AssistantClient` in `sdk/src/assistants.rs` has no `context` or `version` field and its search cannot filter by graph ID. Langstar reads one API key and one workspace at a time, from flags, environment variables or the config file. Langstar exits with status 1 when a command fails, and clap, the library that parses langstar's flags, exits with status 2 when it rejects a mistyped flag or a missing required one. @@ -30,6 +30,10 @@ Export runs with the source environment's credentials and writes a file. Import The LangSmith Go CLI moves a prompt through a file the same way, as `references/langsmith-cli.md` shows. Langstar's export asks for the model, and the Go CLI's pull does not. Langstar's import reads the file export writes, while the Go CLI's push sends whatever JSON it reads as the manifest, so its own pull output reaches LangSmith wrapped in an envelope. Langstar's import also requires `--workspace-id`, so langstar imports into the workspace the user typed on the command line, even when `LANGSMITH_WORKSPACE_ID` names another. +### Assistant commands send their requests to the deployment's `url` + +The control plane returns each deployment's Agent Server address in a top-level `url`, which the SDK's `Deployment` drops today. In [export-import-prompts-and-assistants Task 2.2 Return the URL of a deployment built from GitHub](tasks.md#L13), a developer adds `url` to `Deployment` and has `resolve_deployment_url` use it, falling back to `source_config.custom_url` when `url` is null, missing or empty. Every `langstar assistant` command then reaches a deployment built from GitHub, and the assistant export, compare, verify and import commands are built after that change merges. + ### The export file keeps the manifest LangSmith returned Export stores the manifest as raw JSON. The typed `StructuredPrompt` struct cannot hold a model binding, placeholders, several message types or partial variables, so routing through it would drop them. Langstar reads the manifest through typed views only to fill in `requires`. @@ -81,9 +85,7 @@ Without `--update`, import lists the differences and exits with status 3. `--dry ### Compare reports differences by JSON path -Compare reads the prompt or assistant from the destination, puts it in the shape of the export file's `repo`, `commit` and `assistant` objects, and lists every value that differs from the file. Compare checks every key in those objects, including the `id` arrays inside a manifest that name the prompt and model classes, such as `ChatOpenAI`, and skips `commit.model_config` and `commit.model_provider`, which LangSmith fills in for each workspace. Export leaves the values the server sets, such as IDs, version numbers, timestamps, owner and like counts, out of those objects, so they never show up as differences. The source environment's workspace, deployment and IDs sit in `source`, so they never show up as differences against a copy. Each difference names its JSON Pointer path and both values. For a prompt, the comparison covers the commit description, the commit tags and the commit each one points at, any tag on the compared commit that the file lacks, and the repo's description, readme, tags and visibility as well as the manifest. Import runs the same comparison after it writes, so a field LangSmith or the Agent Server drops shows up as a difference. - -LangSmith fills in `model_config` and `model_provider` itself when import pushes a commit, since no field of the create-commit request holds either one. When the destination fills them in differently from the source, a check on them would fail every prompt import with status 3. Compare checks the model inside the manifest instead, through the chat model constructor's `id` array and its settings. +Compare reads the prompt or assistant from the destination, puts it in the shape of the export file's `repo`, `commit` and `assistant` objects, and lists every value that differs from the file. Compare checks every key in those objects, including the chat model's settings and the `id` arrays inside a manifest that name the prompt and model classes, such as `ChatOpenAI`. It skips `commit.model_config` and `commit.model_provider`, because LangSmith fills both in itself when import pushes a commit, and a destination workspace can fill them in differently from the source. Export leaves the values the server sets, such as IDs, version numbers, timestamps, owner and like counts, out of those objects, so they never show up as differences. The source environment's workspace, deployment and IDs sit in `source`, so they never show up as differences against a copy. Each difference names its JSON Pointer path and both values. For a prompt, the comparison covers the commit description, the commit tags and the commit each one points at, any tag on the compared commit that the file lacks, and the repo's description, readme, tags and visibility as well as the manifest. Import runs the same comparison after it writes, so a field LangSmith or the Agent Server drops shows up as a difference. ### Verify checks what the copy needs to run @@ -119,10 +121,9 @@ A mise task runs a short Python script that pulls the imported prompt with `incl - [LangSmith may store a pushed model differently from the source] → Import reads the prompt back and reports any difference, and in [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8) a developer checks the round trip against the live API before anyone builds the commands. - [Pushing a manifest identical to the latest commit may return 409] → Import compares first and pushes nothing when the prompt matches. -- [langstar finds a deployment's Agent Server URL only in `source_config.custom_url` (`resolve_deployment_url` in `cli/src/deployment_utils.rs`), and a deployment built from GitHub has none, so every `langstar assistant` command, the new ones included, fails on such a deployment] → In [export-import-prompts-and-assistants Task 2.2 Return the URL of a deployment built from GitHub](tasks.md#L13), a developer adds the control plane's top-level `url` to the SDK's `Deployment`, which drops it today, and makes `resolve_deployment_url` use it, before anyone builds the assistant export, compare, verify and import commands. - [The SDK changes for export and import add public fields to `Deployment`, `Prompt`, `CommitManifestResponse`, `CommitData`, `CommitRequest`, `Assistant` and the assistant request structs, so a crate that builds one of those structs with a struct literal or destructures it with a pattern that names every field stops compiling] → Each pull request that adds a field updates every struct literal of that struct in this repository. [Should the SDK field additions count as a breaking change?](#9---should-the-sdk-field-additions-count-as-a-breaking-change) asks codekiln whether those pull requests also carry a `BREAKING CHANGE:` footer, which `docs/dev/git-scm-conventions.md` says makes the release a major version. - [A prompt's manifest may hold a credential written as a literal model setting, rather than as a secret reference] → Export refuses with status 1 and writes nothing when a chat model setting whose name ends in `api_key` holds a string, and names that setting's path without its value. A credential under any other name, such as a header, still reaches the file, because langstar cannot tell it from any other string. The prompt export docs in [export-import-prompts-and-assistants Task 5.2 Document prompt export](tasks.md#L54) tell users to keep credentials in the workspace's secrets and to read an export file before committing it. -- [An assistant's configuration, context or metadata may hold a credential, which export copies into a file meant for Git] → Langstar cannot tell a credential from any other string in an assistant's configuration, as it cannot tell which key holds a prompt name, so export writes those objects as the Agent Server returns them. The assistant export docs in [export-import-prompts-and-assistants Task 8.2 Document assistant export](tasks.md#L106) tell users to keep credentials in the deployment's environment variables or the workspace's secrets rather than in an assistant, and to read an export file before committing it. +- [An assistant's configuration, context or metadata may hold a credential, which export copies into a file meant for Git] → Export writes those objects as the Agent Server returns them, since a credential there looks like any other string. The assistant export docs in [export-import-prompts-and-assistants Task 8.2 Document assistant export](tasks.md#L106) tell users to keep credentials in the deployment's environment variables or the workspace's secrets, and to read an export file before committing it. - [A tag in the destination may already point at another commit] → `--update` moves the tag, and `--dry-run` lists the move. - [`--update` deletes a tag on the matching destination commit that the file lacks, and a graph on a deployment may read that tag] → `--dry-run` lists each delete, and `assistant verify` reports a prompt tag an assistant refers to that the workspace no longer has. - [The control plane may refuse a second deployment from this repository with 409] → In [export-import-prompts-and-assistants Task 2.3 Register the test graph under a second graph ID](tasks.md#L18), a developer deploys the new `langgraph.json` once, before anyone builds the fixture. If the control plane answers with 409, the developer brings [Which second workspace and second deployment should the tests use?](#5---which-second-workspace-and-second-deployment-should-the-tests-use) back to codekiln. diff --git a/openspec/changes/export-import-prompts-and-assistants/references/langgraph-sdk.md b/openspec/changes/export-import-prompts-and-assistants/references/langgraph-sdk.md index a5d20229e..29d29adbf 100644 --- a/openspec/changes/export-import-prompts-and-assistants/references/langgraph-sdk.md +++ b/openspec/changes/export-import-prompts-and-assistants/references/langgraph-sdk.md @@ -4,7 +4,7 @@ Read at [langgraph 8a77332d](https://github.com/langchain-ai/langgraph/tree/8a77 [`create`](https://github.com/langchain-ai/langgraph/blob/8a77332d0db75c49d296b1f3833a3a5ccd5f0dc8/libs/sdk-py/langgraph_sdk/_async/assistants.py#L314) posts `graph_id`, `config`, `context`, `metadata`, `assistant_id`, `if_exists` (`raise` or `do_nothing`), `name` and `description` to `/assistants`. -[`update`](https://github.com/langchain-ai/langgraph/blob/8a77332d0db75c49d296b1f3833a3a5ccd5f0dc8/libs/sdk-py/langgraph_sdk/_async/assistants.py#L385) patches the same fields; the server merges `metadata` into the existing metadata. [`get_versions`](https://github.com/langchain-ai/langgraph/blob/8a77332d0db75c49d296b1f3833a3a5ccd5f0dc8/libs/sdk-py/langgraph_sdk/_async/assistants.py#L656) lists the versions updates create. +[`update`](https://github.com/langchain-ai/langgraph/blob/8a77332d0db75c49d296b1f3833a3a5ccd5f0dc8/libs/sdk-py/langgraph_sdk/_async/assistants.py#L385) patches `graph_id`, `config`, `context`, `metadata`, `name` and `description`, and the server merges `metadata` into the existing metadata. [`get_versions`](https://github.com/langchain-ai/langgraph/blob/8a77332d0db75c49d296b1f3833a3a5ccd5f0dc8/libs/sdk-py/langgraph_sdk/_async/assistants.py#L656) lists the versions updates create. [`AssistantBase`](https://github.com/langchain-ai/langgraph/blob/8a77332d0db75c49d296b1f3833a3a5ccd5f0dc8/libs/sdk-py/langgraph_sdk/schema.py#L246) returns `assistant_id`, `graph_id`, `config`, `context`, `created_at`, `metadata`, `version`, `name` and `description`. `context` sits beside `config.configurable`; newer graphs read it and older ones read `configurable`. From 8f2c48db5a9438c66b38723c87d7eafaed309df4 Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Wed, 14 Oct 2026 01:41:40 -0400 Subject: [PATCH 49/90] =?UTF-8?q?=F0=9F=93=9A=20docs(openspec):=20test=20m?= =?UTF-8?q?ocked=20commands=20as=20unit=20tests=20through=20mockito=20and?= =?UTF-8?q?=20with=5Fbase=5Furls,=20rerun=20the=20stale=20prompt=20endpoin?= =?UTF-8?q?t=20extract,=20split=20the=20group=203=20introduction,=20and=20?= =?UTF-8?q?list=20tenant=5Fid=20among=20the=20new=20Prompt=20fields?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit #807 Export, import, compare, and verify prompts and assistants across LangSmith environments Co-Authored-By: Claude Opus 5.5 --- .../design.md | 4 ++-- .../proposal.md | 2 +- .../tasks.md | 22 ++++++++++--------- 3 files changed, 15 insertions(+), 13 deletions(-) diff --git a/openspec/changes/export-import-prompts-and-assistants/design.md b/openspec/changes/export-import-prompts-and-assistants/design.md index c1f8f0eac..80be0df39 100644 --- a/openspec/changes/export-import-prompts-and-assistants/design.md +++ b/openspec/changes/export-import-prompts-and-assistants/design.md @@ -122,8 +122,8 @@ A mise task runs a short Python script that pulls the imported prompt with `incl - [LangSmith may store a pushed model differently from the source] → Import reads the prompt back and reports any difference, and in [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8) a developer checks the round trip against the live API before anyone builds the commands. - [Pushing a manifest identical to the latest commit may return 409] → Import compares first and pushes nothing when the prompt matches. - [The SDK changes for export and import add public fields to `Deployment`, `Prompt`, `CommitManifestResponse`, `CommitData`, `CommitRequest`, `Assistant` and the assistant request structs, so a crate that builds one of those structs with a struct literal or destructures it with a pattern that names every field stops compiling] → Each pull request that adds a field updates every struct literal of that struct in this repository. [Should the SDK field additions count as a breaking change?](#9---should-the-sdk-field-additions-count-as-a-breaking-change) asks codekiln whether those pull requests also carry a `BREAKING CHANGE:` footer, which `docs/dev/git-scm-conventions.md` says makes the release a major version. -- [A prompt's manifest may hold a credential written as a literal model setting, rather than as a secret reference] → Export refuses with status 1 and writes nothing when a chat model setting whose name ends in `api_key` holds a string, and names that setting's path without its value. A credential under any other name, such as a header, still reaches the file, because langstar cannot tell it from any other string. The prompt export docs in [export-import-prompts-and-assistants Task 5.2 Document prompt export](tasks.md#L54) tell users to keep credentials in the workspace's secrets and to read an export file before committing it. -- [An assistant's configuration, context or metadata may hold a credential, which export copies into a file meant for Git] → Export writes those objects as the Agent Server returns them, since a credential there looks like any other string. The assistant export docs in [export-import-prompts-and-assistants Task 8.2 Document assistant export](tasks.md#L106) tell users to keep credentials in the deployment's environment variables or the workspace's secrets, and to read an export file before committing it. +- [A prompt's manifest may hold a credential written as a literal model setting, rather than as a secret reference] → Export refuses with status 1 and writes nothing when a chat model setting whose name ends in `api_key` holds a string, and names that setting's path without its value. A credential under any other name, such as a header, still reaches the file, because langstar cannot tell it from any other string. The prompt export docs in [export-import-prompts-and-assistants Task 5.2 Document prompt export](tasks.md#L56) tell users to keep credentials in the workspace's secrets and to read an export file before committing it. +- [An assistant's configuration, context or metadata may hold a credential, which export copies into a file meant for Git] → Export writes those objects as the Agent Server returns them, since a credential there looks like any other string. The assistant export docs in [export-import-prompts-and-assistants Task 8.2 Document assistant export](tasks.md#L108) tell users to keep credentials in the deployment's environment variables or the workspace's secrets, and to read an export file before committing it. - [A tag in the destination may already point at another commit] → `--update` moves the tag, and `--dry-run` lists the move. - [`--update` deletes a tag on the matching destination commit that the file lacks, and a graph on a deployment may read that tag] → `--dry-run` lists each delete, and `assistant verify` reports a prompt tag an assistant refers to that the workspace no longer has. - [The control plane may refuse a second deployment from this repository with 409] → In [export-import-prompts-and-assistants Task 2.3 Register the test graph under a second graph ID](tasks.md#L18), a developer deploys the new `langgraph.json` once, before anyone builds the fixture. If the control plane answers with 409, the developer brings [Which second workspace and second deployment should the tests use?](#5---which-second-workspace-and-second-deployment-should-the-tests-use) back to codekiln. diff --git a/openspec/changes/export-import-prompts-and-assistants/proposal.md b/openspec/changes/export-import-prompts-and-assistants/proposal.md index c6b38259a..4e44489bd 100644 --- a/openspec/changes/export-import-prompts-and-assistants/proposal.md +++ b/openspec/changes/export-import-prompts-and-assistants/proposal.md @@ -30,7 +30,7 @@ A prompt belongs to a workspace and an assistant belongs to a deployment, but bo ## Impact -- SDK: `sdk/src/prompts.rs` gains a commit read that asks for the model and returns it on `CommitManifestResponse`, a commit description on commit create, the new commit's `id`, the repo's readme and tags on `Prompt`, a repo update, a commit list that returns each commit's `id`, calls to list, create and move commit tags, and a tag delete that `prompt import --update` and the end-to-end test use. `sdk/src/assistants.rs` gains the `context` and `version` fields, `assistant_id` and `if_exists` on create, `graph_id` on update, and a search by graph ID. `sdk/src/deployments.rs` gains the deployment's `url`. Each new public field breaks a crate outside this repository that builds that SDK struct with a struct literal or destructures it with a pattern that names every field; codekiln decides in [Should the SDK field additions count as a breaking change?](design.md#9---should-the-sdk-field-additions-count-as-a-breaking-change) whether that makes the release a major version. +- SDK: `sdk/src/prompts.rs` gains a commit read that asks for the model and returns it on `CommitManifestResponse`, a commit description on commit create, the new commit's `id`, the repo's readme, tags and `tenant_id` on `Prompt`, from which export fills `source.workspace_id`, a repo update, a commit list that returns each commit's `id`, calls to list, create and move commit tags, and a tag delete that `prompt import --update` and the end-to-end test use. `sdk/src/assistants.rs` gains the `context` and `version` fields, `assistant_id` and `if_exists` on create, `graph_id` on update, and a search by graph ID. `sdk/src/deployments.rs` gains the deployment's `url`. Each new public field breaks a crate outside this repository that builds that SDK struct with a struct literal or destructures it with a pattern that names every field; codekiln decides in [Should the SDK field additions count as a breaking change?](design.md#9---should-the-sdk-field-additions-count-as-a-breaking-change) whether that makes the release a major version. - CLI: new subcommands in `cli/src/commands/prompt.rs` and `cli/src/commands/assistant.rs`, and a shared module for the export file, the destination record, comparison and the exit status. `resolve_deployment_url` in `cli/src/deployment_utils.rs` also returns the Agent Server URL of a deployment built from GitHub, which every `langstar assistant` command needs. - Tests need a second workspace, a second deployment and one model provider key for the structured output check. codekiln creates the workspace and the key, and langstar's tests read them through `fnox.toml` and the CI workflow; the destination deployment fixture from [export-import-prompts-and-assistants Task 2.4 Add a destination deployment fixture](tasks.md#L19) creates the deployment whenever the cleanup workflow has deleted it, as the source fixture does today. Nobody knows yet whether the control plane accepts a second deployment from this repository. In [export-import-prompts-and-assistants Task 2.3 Register the test graph under a second graph ID](tasks.md#L18), a developer deploys the new `langgraph.json` once to find out, before anyone builds the fixture. - Docs: a new usage page on moving prompts and assistants between environments. diff --git a/openspec/changes/export-import-prompts-and-assistants/tasks.md b/openspec/changes/export-import-prompts-and-assistants/tasks.md index 5c939cb4f..5dfb6635a 100644 --- a/openspec/changes/export-import-prompts-and-assistants/tasks.md +++ b/openspec/changes/export-import-prompts-and-assistants/tasks.md @@ -4,7 +4,7 @@ Each numbered group below is one pull request, merged in the order listed, excep ## 1. Check the API round trip before building on it -- [ ] 1.1 Extract the commit get, commit list (`GET /api/v1/commits/{owner}/{repo}`), commit create, repo update (`PATCH /api/v1/repos/{owner}/{repo}`), repo tags (`/api/v1/repos/{owner}/{repo}/tags` and `/api/v1/repos/{owner}/{repo}/tags/{tag_name}`), workspace secrets and playground settings endpoints from `reference/openapi/langchain/langsmith/openapi.json` into `reference/api-specs/langsmith/` and add their jq queries to `FRAGMENTS.md`; verify by rerunning each query and getting the same file +- [ ] 1.1 Rerun the jq query for `prompt-endpoints.json` in `reference/api-specs/langsmith/FRAGMENTS.md`, so the extract picks up the `include` parameter of the commit read that its 2025-11-29 copy lacks, and extract the workspace secrets endpoints (`/api/v1/workspaces/current/secrets`) from `reference/openapi/langchain/langsmith/openapi.json` into a new file in `reference/api-specs/langsmith/` with its jq query in `FRAGMENTS.md`; verify by rerunning each query and getting the same file - [ ] 1.2 Write `reference/research/807-openapi-validation.md`, recording from the live API: the manifest returned with `include=model` for a structured prompt with a model, the outer `id` of that manifest, the `model_config` and `model_provider` values returned beside it, whether pushing that manifest to a second workspace returns the same manifest, `model_config` and `model_provider`, whether `model_provider` matches the class name export writes to `requires.model_class`, what pushing an identical manifest returns, what the Agent Server returns for `context`, whether an Agent Server assistant update that sends a metadata key with a null value removes that key, whether an update that sends `config` or `context` replaces the assistant's object or merges into it, whether an update can clear an assistant's description, and what `POST /api/v1/prompts/invoke_prompt` returns for a prompt with a model; verify that every claim in the report names the request that produced it ## 2. Test environments @@ -20,12 +20,14 @@ Each numbered group below is one pull request, merged in the order listed, excep ## 3. SDK reads and writes the fields export and import need -Each task in this group adds a public field to one of these SDK structs: `Prompt` (the struct `PromptClient::get` returns, built with a struct literal in `test_prompt_serialization` in `sdk/src/prompts.rs`), `CommitManifestResponse` (the struct `PromptClient::get_commit` returns), `CommitData`, `CommitRequest`, `Assistant`, `CreateAssistantRequest`, `UpdateAssistantRequest` and `AssistantSearchRequest`. A crate that builds one of them with a struct literal, or destructures one with a pattern that names every field, stops compiling; nothing in this repository destructures them that way, so each task updates every struct literal of its struct in this repository. Whether this group's pull request and task 2.2's, which adds `url` to `Deployment`, carry a `BREAKING CHANGE:` footer waits on [Should the SDK field additions count as a breaking change?](design.md#9---should-the-sdk-field-additions-count-as-a-breaking-change). Each integration test in this group carries `#[cfg_attr(not(feature = "integration-tests"), ignore)]`, as the tests in `sdk/tests/structured_prompts_integration_test.rs` do, and not a bare `#[ignore]`. The SDK Integration Tests job in `.github/workflows/ci.yml` and `mise run check` both turn that feature on, so both run the test, while a bare `#[ignore]` would skip it in both. +Each task in this group adds public fields to SDK structs and updates every struct literal of those structs in this repository, so the workspace still compiles. The structs are `Prompt`, `CommitManifestResponse`, `CommitData`, `CommitRequest`, `Assistant`, `CreateAssistantRequest`, `UpdateAssistantRequest` and `AssistantSearchRequest`. A crate outside this repository that builds one of them with a struct literal stops compiling too, and codekiln decides in [Should the SDK field additions count as a breaking change?](design.md#9---should-the-sdk-field-additions-count-as-a-breaking-change) whether this group's pull request and task 2.2's carry a `BREAKING CHANGE:` footer. -- [ ] 3.1 Add `description`, `model_config` and `model_provider` to `CommitManifestResponse`, and a commit read that sends `include=model` and returns that struct, to `sdk/src/prompts.rs`; verify with an httpmock test that asserts the query string and the parsed fields, and an integration test that reads a prompt with a model -- [ ] 3.2 Add the commit's `id` to `CommitData` in `sdk/src/prompts.rs`, a call that lists a prompt's commits with their `id` and hash (`GET /api/v1/commits/{owner}/{repo}`), calls to list, create and move commit tags, which name their commit by that `id` in `POST /api/v1/repos/{owner}/{repo}/tags` and `PATCH /api/v1/repos/{owner}/{repo}/tags/{tag_name}`, and a call that deletes a tag with `DELETE /api/v1/repos/{owner}/{repo}/tags/{tag_name}`, which `prompt import --update` from 7.2 and the end-to-end test in 10.3 use; verify with httpmock tests that assert `commit_id` in each create and move request body and the path of the delete request, and an integration test that pushes two commits, tags the first, finds the second commit's `id` through the commit list, moves the tag to it, reads the tag back through the SDK, deletes it, and asserts that the tag list no longer holds it -- [ ] 3.3 Add `context` and `version` to `Assistant`, `context` and `graph_id` to `UpdateAssistantRequest`, `context`, `assistant_id` and `if_exists` to `CreateAssistantRequest`, and a `graph_id` filter to `AssistantSearchRequest`, in `sdk/src/assistants.rs`; verify with httpmock tests of the request bodies and an integration test that creates an assistant with context on the destination deployment from 2.4, reads it back with its version, finds it by graph ID, moves it from `test_graph` to `test_graph_alt`, and reads `test_graph_alt` back. The test runs on the destination deployment because the source deployment serves only `test_graph`, and task 2.4, which adds the destination fixture, merges before this group. -- [ ] 3.4 Add `description` to `CommitRequest`, and `readme`, the repo's `tags` and `tenant_id` to `Prompt`, which `PromptClient::get` returns and export reads the repo details and `source.workspace_id` from, and a repo update that sends `PATCH /api/v1/repos/{owner}/{repo}` with `description`, `readme`, `tags` and `is_public`, to `sdk/src/prompts.rs`; verify with httpmock tests of the request bodies and of a `PromptClient::get` response that holds `readme`, `tags` and `tenant_id`, and an integration test that pushes a commit with a description and reads it back through the commit read from 3.1, then updates the repo's description, readme, tags and visibility and reads each back through `PromptClient::get` +Each integration test in this group carries `#[cfg_attr(not(feature = "integration-tests"), ignore)]`, as the tests in `sdk/tests/structured_prompts_integration_test.rs` do, so that the SDK Integration Tests job in `.github/workflows/ci.yml` and `mise run check` both run it. + +- [ ] 3.1 Add `description`, `model_config` and `model_provider` to `CommitManifestResponse`, and a commit read that sends `include=model` and returns that struct, to `sdk/src/prompts.rs`; verify with a `mockito` test that asserts the query string and the parsed fields, and an integration test that reads a prompt with a model +- [ ] 3.2 Add the commit's `id` to `CommitData` in `sdk/src/prompts.rs`, a call that lists a prompt's commits with their `id` and hash (`GET /api/v1/commits/{owner}/{repo}`), calls to list, create and move commit tags, which name their commit by that `id` in `POST /api/v1/repos/{owner}/{repo}/tags` and `PATCH /api/v1/repos/{owner}/{repo}/tags/{tag_name}`, and a call that deletes a tag with `DELETE /api/v1/repos/{owner}/{repo}/tags/{tag_name}`, which `prompt import --update` from 7.2 and the end-to-end test in 10.3 use; verify with `mockito` tests that assert `commit_id` in each create and move request body and the path of the delete request, and an integration test that pushes two commits, tags the first, finds the second commit's `id` through the commit list, moves the tag to it, reads the tag back through the SDK, deletes it, and asserts that the tag list no longer holds it +- [ ] 3.3 Add `context` and `version` to `Assistant`, `context` and `graph_id` to `UpdateAssistantRequest`, `context`, `assistant_id` and `if_exists` to `CreateAssistantRequest`, and a `graph_id` filter to `AssistantSearchRequest`, in `sdk/src/assistants.rs`; verify with `mockito` tests of the request bodies and an integration test that creates an assistant with context on the destination deployment from 2.4, reads it back with its version, finds it by graph ID, moves it from `test_graph` to `test_graph_alt`, and reads `test_graph_alt` back. The test runs on the destination deployment because the source deployment serves only `test_graph`, and task 2.4, which adds the destination fixture, merges before this group. +- [ ] 3.4 Add `description` to `CommitRequest`, and `readme`, the repo's `tags` and `tenant_id` to `Prompt`, which `PromptClient::get` returns and export reads the repo details and `source.workspace_id` from, and a repo update that sends `PATCH /api/v1/repos/{owner}/{repo}` with `description`, `readme`, `tags` and `is_public`, to `sdk/src/prompts.rs`; verify with `mockito` tests of the request bodies and of a `PromptClient::get` response that holds `readme`, `tags` and `tenant_id`, and an integration test that pushes a commit with a description and reads it back through the commit read from 3.1, then updates the repo's description, readme, tags and visibility and reads each back through `PromptClient::get` ## 4. Shared code for the prompt and assistant commands: the export file, the comparison and the report @@ -49,8 +51,8 @@ Each task in this group adds a public field to one of these SDK structs: `Prompt - An error test that exports a handle the workspace has no prompt for, and asserts that it exits with status 1, names the handle and writes no file. - A CLI test that runs `prompt export` without `--output`, parses all of stdout as one JSON document, and asserts that its `source.commit_hash` matches the SDK's read and that any progress message is on stderr. - An integration test that gives one commit two tags, exports it twice, and asserts that the two files are identical. - - A CLI test that serves, through httpmock, a commit whose `ChatOpenAI` constructor holds a string under `openai_api_key`, runs `prompt export my-prompt --output my-prompt.json`, and asserts that it exits with status 1, writes no file and nothing to stdout, and names the setting's path on stderr without the string. - - A CLI test that serves a prompt through httpmock, runs `prompt export my-prompt` with no `--workspace-id` and no `LANGSMITH_WORKSPACE_ID`, and asserts that `source.workspace_id` holds the repo's `tenant_id`. + - A unit test in `cli/src/commands/prompt.rs` that serves, through a `mockito` server and `LangchainClient::with_base_urls`, a commit whose `ChatOpenAI` constructor holds a string under `openai_api_key`, calls the export code with `--output my-prompt.json`, and asserts that it returns an error that exits with status 1 and names the setting's path without the string, and that it writes no file. + - A unit test in `cli/src/commands/prompt.rs` that serves a prompt through a `mockito` server and `LangchainClient::with_base_urls`, calls the export code with no workspace ID set, and asserts that `source.workspace_id` holds the repo's `tenant_id`. - [ ] 5.2 Document `prompt export` in `docs/usage/` and the README. The page says that export refuses a prompt whose model holds an API key as a literal string, names that setting and writes no file, and that a credential kept under any other name, such as a header, still reaches the file; tells users to keep credentials in the workspace's secrets; and tells them to read an export file before they commit it. Verify that each documented command runs as written. ## 6. prompt compare and verify @@ -90,7 +92,7 @@ Each task in this group adds a public field to one of these SDK structs: `Prompt - An integration test that `--update` against a destination whose `prod` tag points at an earlier commit while the manifest matches moves the tag and adds no commit, and that `--dry-run` lists the move. - An integration test that `--update` against a destination whose latest commit matches the file and carries a `staging` tag the file lacks deletes `staging`, adds no commit and exits with status 0, and that `--dry-run` lists the delete. - An integration test that `--dry-run` writes nothing. - - An httpmock test in which the commit read back lacks a field the file holds, asserting that import names that field's JSON Pointer path, exits with status 3, and leaves the new commit hash in the destination record. + - A unit test in `cli/src/commands/prompt.rs` that serves, through a `mockito` server and `LangchainClient::with_base_urls`, a commit read back that lacks a field the file holds, calls the import code, and asserts that the result names that field's JSON Pointer path and gives exit status 3, and that the destination record holds the new commit hash. - [ ] 7.3 Add the structured output check as a mise task that pulls the imported prompt with `include_model=True` and a `secrets` map holding the model provider key read from the environment, invokes it and validates the result against the schema; verify by having the import integration test create a `json_schema` prompt and a `function_calling` prompt, each saved with a small OpenAI model, import both into the destination workspace, and run the check on each - [ ] 7.4 Document prompt import, the workspace scope of prompts and setting up destination credentials; verify that each documented command runs as written @@ -126,7 +128,7 @@ Each task in this group adds a public field to one of these SDK structs: `Prompt - An integration test that creates a tagged prompt in the test workspace, imports an assistant that refers to it into the destination deployment, and checks the name, graph ID, configuration, context, metadata and prompt references through the Agent Server API. - An integration test that `--record ` writes the destination record to that path. - Integration tests that a file whose graph ID the deployment lacks, and a file with a missing prompt tag, each create no assistant and exit with status 3. - - An httpmock test in which the assistant read back differs from the file, asserting that import exits with status 3, that the destination record holds the new assistant ID, and that a second import creates no assistant. + - A unit test in `cli/src/commands/assistant.rs` that serves, through a `mockito` server and `LangchainClient::with_base_urls`, an assistant read back that differs from the file, calls the import code twice, and asserts that the first call gives exit status 3 and leaves the new assistant ID in the destination record, and that the second call creates no assistant. - A CLI test that `assistant import` of a prompt export exits with status 1, names `kind` and the value `prompt`, and creates no assistant. - An integration test that creates an assistant with a chosen ID on the destination deployment, writes a destination record that holds that ID as pending, imports the file, and asserts that the deployment holds one assistant from the file, with that ID, and that the record marks the ID created. - [ ] 10.2 Add `--update`, `--assistant-id` and `--dry-run`. Verify with these tests: From 2ca8e6ca324e54f3793ed198f8e7e2892a6207df Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Wed, 14 Oct 2026 01:47:21 -0400 Subject: [PATCH 50/90] =?UTF-8?q?=F0=9F=93=9A=20docs(openspec):=20test=20t?= =?UTF-8?q?hat=20prompt=20verify=20and=20assistant=20verify=20report=20a?= =?UTF-8?q?=20changed=20field=20as=20compare=20does?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit #807 Export, import, compare, and verify prompts and assistants across LangSmith environments Co-Authored-By: Claude Opus 5.5 --- openspec/changes/export-import-prompts-and-assistants/design.md | 2 +- openspec/changes/export-import-prompts-and-assistants/tasks.md | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/openspec/changes/export-import-prompts-and-assistants/design.md b/openspec/changes/export-import-prompts-and-assistants/design.md index 80be0df39..f8c45a976 100644 --- a/openspec/changes/export-import-prompts-and-assistants/design.md +++ b/openspec/changes/export-import-prompts-and-assistants/design.md @@ -123,7 +123,7 @@ A mise task runs a short Python script that pulls the imported prompt with `incl - [Pushing a manifest identical to the latest commit may return 409] → Import compares first and pushes nothing when the prompt matches. - [The SDK changes for export and import add public fields to `Deployment`, `Prompt`, `CommitManifestResponse`, `CommitData`, `CommitRequest`, `Assistant` and the assistant request structs, so a crate that builds one of those structs with a struct literal or destructures it with a pattern that names every field stops compiling] → Each pull request that adds a field updates every struct literal of that struct in this repository. [Should the SDK field additions count as a breaking change?](#9---should-the-sdk-field-additions-count-as-a-breaking-change) asks codekiln whether those pull requests also carry a `BREAKING CHANGE:` footer, which `docs/dev/git-scm-conventions.md` says makes the release a major version. - [A prompt's manifest may hold a credential written as a literal model setting, rather than as a secret reference] → Export refuses with status 1 and writes nothing when a chat model setting whose name ends in `api_key` holds a string, and names that setting's path without its value. A credential under any other name, such as a header, still reaches the file, because langstar cannot tell it from any other string. The prompt export docs in [export-import-prompts-and-assistants Task 5.2 Document prompt export](tasks.md#L56) tell users to keep credentials in the workspace's secrets and to read an export file before committing it. -- [An assistant's configuration, context or metadata may hold a credential, which export copies into a file meant for Git] → Export writes those objects as the Agent Server returns them, since a credential there looks like any other string. The assistant export docs in [export-import-prompts-and-assistants Task 8.2 Document assistant export](tasks.md#L108) tell users to keep credentials in the deployment's environment variables or the workspace's secrets, and to read an export file before committing it. +- [An assistant's configuration, context or metadata may hold a credential, which export copies into a file meant for Git] → Export writes those objects as the Agent Server returns them, since a credential there looks like any other string. The assistant export docs in [export-import-prompts-and-assistants Task 8.2 Document assistant export](tasks.md#L109) tell users to keep credentials in the deployment's environment variables or the workspace's secrets, and to read an export file before committing it. - [A tag in the destination may already point at another commit] → `--update` moves the tag, and `--dry-run` lists the move. - [`--update` deletes a tag on the matching destination commit that the file lacks, and a graph on a deployment may read that tag] → `--dry-run` lists each delete, and `assistant verify` reports a prompt tag an assistant refers to that the workspace no longer has. - [The control plane may refuse a second deployment from this repository with 409] → In [export-import-prompts-and-assistants Task 2.3 Register the test graph under a second graph ID](tasks.md#L18), a developer deploys the new `langgraph.json` once, before anyone builds the fixture. If the control plane answers with 409, the developer brings [Which second workspace and second deployment should the tests use?](#5---which-second-workspace-and-second-deployment-should-the-tests-use) back to codekiln. diff --git a/openspec/changes/export-import-prompts-and-assistants/tasks.md b/openspec/changes/export-import-prompts-and-assistants/tasks.md index 5dfb6635a..ee5bdec64 100644 --- a/openspec/changes/export-import-prompts-and-assistants/tasks.md +++ b/openspec/changes/export-import-prompts-and-assistants/tasks.md @@ -67,6 +67,7 @@ Each integration test in this group carries `#[cfg_attr(not(feature = "integrati - A CLI test that `prompt compare` of a file whose handle the workspace lacks exits with status 1 and names the handle. - [ ] 6.2 Add `langstar prompt verify` with the secret check and the saved model configuration note, which matches `requires.model_class` against the class name at the end of each configuration's settings `id`. Verify with these tests: - An integration test against a workspace that lacks a secret the file lists, asserting that verify names the secret and exits with status 3, and one against a workspace that has every listed secret, asserting that verify exits with status 0. + - An integration test that exports a prompt, changes its model temperature through the SDK, runs verify, and asserts that it reports the path and both values as `prompt compare` does and exits with status 3. - A test that a missing saved model configuration is reported as a note and leaves the exit status at 0. - A test that verify of a prompt without a model, whose `requires.model_class` is null, reports no saved model configuration note and exits with status 0. - A test that a file whose manifest refers to a secret its `requires.secrets` leaves out exits with status 1 and names `requires.secrets`. @@ -116,6 +117,7 @@ Each integration test in this group carries `#[cfg_attr(not(feature = "integrati - A unit test that a file whose configuration names another prompt at a pointer under `requires.prompts` exits with status 1 and names the pointer and both values. - [ ] 9.2 Add `langstar assistant verify` with `--assistant-id`, `--workspace-id` for the workspace that holds the prompts, the graph check, and the prompt check, which reads `GET /api/v1/commits/{owner}/{repo}/{tag}`. Verify with these tests: - An integration test that runs verify with `--assistant-id` against an exported assistant and exits with status 0. + - An integration test that exports an assistant, changes its metadata through the SDK, runs verify with `--assistant-id` set to that assistant, and asserts the reported path and exit status 3. - An integration test of a file whose graph ID the deployment lacks, which reports the graph ID and exits with status 3. - An integration test of a file whose prompt tag the workspace lacks, which reports the prompt and exits with status 3. - An integration test that runs verify with `--workspace-id` set to the destination workspace, for an assistant whose referenced prompt tag exists only in that workspace, and asserts that verify exits with status 0 and that `checks` reports the prompt as found. From 332d3abc663b1a3eb454fe77fb99c118656f141a Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Wed, 14 Oct 2026 01:59:58 -0400 Subject: [PATCH 51/90] =?UTF-8?q?=F0=9F=93=9A=20docs(openspec):=20point=20?= =?UTF-8?q?the=20langgraph=20get=20docstring=20link=20at=20the=20created?= =?UTF-8?q?=5Fby=20line?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit #807 Export, import, compare, and verify prompts and assistants across LangSmith environments Co-Authored-By: Claude Opus 5.5 --- .../references/langgraph-sdk.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/openspec/changes/export-import-prompts-and-assistants/references/langgraph-sdk.md b/openspec/changes/export-import-prompts-and-assistants/references/langgraph-sdk.md index 29d29adbf..8e3c07697 100644 --- a/openspec/changes/export-import-prompts-and-assistants/references/langgraph-sdk.md +++ b/openspec/changes/export-import-prompts-and-assistants/references/langgraph-sdk.md @@ -8,6 +8,6 @@ Read at [langgraph 8a77332d](https://github.com/langchain-ai/langgraph/tree/8a77 [`AssistantBase`](https://github.com/langchain-ai/langgraph/blob/8a77332d0db75c49d296b1f3833a3a5ccd5f0dc8/libs/sdk-py/langgraph_sdk/schema.py#L246) returns `assistant_id`, `graph_id`, `config`, `context`, `created_at`, `metadata`, `version`, `name` and `description`. `context` sits beside `config.configurable`; newer graphs read it and older ones read `configurable`. -There is no endpoint that lists graphs. [`search`](https://github.com/langchain-ai/langgraph/blob/8a77332d0db75c49d296b1f3833a3a5ccd5f0dc8/libs/sdk-py/langgraph_sdk/_async/assistants.py#L528) filters by `graph_id`, and every graph has a default assistant with `metadata.created_by` set to `system`, as the [`get` docstring](https://github.com/langchain-ai/langgraph/blob/8a77332d0db75c49d296b1f3833a3a5ccd5f0dc8/libs/sdk-py/langgraph_sdk/_async/assistants.py#L64) shows. +There is no endpoint that lists graphs. [`search`](https://github.com/langchain-ai/langgraph/blob/8a77332d0db75c49d296b1f3833a3a5ccd5f0dc8/libs/sdk-py/langgraph_sdk/_async/assistants.py#L528) filters by `graph_id`, and every graph has a default assistant with `metadata.created_by` set to `system`, as the [`get` docstring](https://github.com/langchain-ai/langgraph/blob/8a77332d0db75c49d296b1f3833a3a5ccd5f0dc8/libs/sdk-py/langgraph_sdk/_async/assistants.py#L81) shows. What langstar takes from it: the fields an assistant export carries, including `context`, and the graph check through search. From 3e07d9797eb0d816a89be12a13b0c53d02ef91fb Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Wed, 14 Oct 2026 02:08:39 -0400 Subject: [PATCH 52/90] =?UTF-8?q?=F0=9F=93=9A=20docs(openspec):=20install?= =?UTF-8?q?=20the=20structured=20output=20check's=20Python=20packages=20th?= =?UTF-8?q?rough=20uv,=20name=20the=20missing-URL=20error=20in=20task=202.?= =?UTF-8?q?2,=20and=20split=20the=20SDK=20impact=20and=20first=20goal?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit #807 Export, import, compare, and verify prompts and assistants across LangSmith environments Co-Authored-By: Claude Opus 5.5 --- .../changes/export-import-prompts-and-assistants/design.md | 6 +++--- .../export-import-prompts-and-assistants/proposal.md | 2 +- .../changes/export-import-prompts-and-assistants/tasks.md | 4 ++-- 3 files changed, 6 insertions(+), 6 deletions(-) diff --git a/openspec/changes/export-import-prompts-and-assistants/design.md b/openspec/changes/export-import-prompts-and-assistants/design.md index f8c45a976..cf3d2b141 100644 --- a/openspec/changes/export-import-prompts-and-assistants/design.md +++ b/openspec/changes/export-import-prompts-and-assistants/design.md @@ -12,7 +12,7 @@ Langstar reads one API key and one workspace at a time, from flags, environment **Goals:** -- Carry a prompt's manifest and model, its commit description and tags, and its repo details, and an assistant's name, description, graph ID, configuration, context and metadata, through a file and back without loss. +- Recreate a prompt or an assistant from its export file with nothing lost. A prompt keeps its manifest and model, its commit description and tags, and its repo details. An assistant keeps its name, description, graph ID, configuration, context and metadata. - Make a repeated import of the same file safe: it finds its earlier copy and changes nothing until the user asks. - Give users one way to check a copy: the same report shape and exit statuses for both resource types. @@ -111,7 +111,7 @@ Status 0 means the resource matches and every check passed. Status 3 means langs ### A Python script checks that an imported structured prompt still returns structured data -A mise task runs a short Python script that pulls the imported prompt with `include_model=True`, invokes it, and checks that the result matches the prompt's schema. The script passes `secrets={"OPENAI_API_KEY": }` to [`pull_prompt`](https://github.com/langchain-ai/langsmith-sdk/blob/47c553caf08c1989e2be3ff1886da3eb801f709f/python/langsmith/client.py#L10228), because the LangSmith Python SDK reads no secret from the environment unless asked and raises when a manifest holds a secret reference and no `secrets` map arrives. The script exists because the LangSmith API has no documented way to run a prompt with its model: the one prompt endpoint in the OpenAPI spec, `POST /api/v1/prompts/invoke_prompt`, takes message templates and inputs with no model or schema, and no LangSmith SDK calls it. In [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8), a developer calls that endpoint on the live API in case it does more than the spec says. The script needs an OpenAI API key. codekiln creates it and stores it in 1Password, in the repository's CI secrets and as the destination workspace's `OPENAI_API_KEY` secret, and in [export-import-prompts-and-assistants Task 2.1 Add the destination workspace and the model provider key](tasks.md#L12) a developer adds it to `fnox.toml` and the CI workflow. +A mise task runs a short Python script with `uv run --script`, which installs the `langsmith` and `langchain-openai` packages the script names as inline dependencies, locally and in the Integration Tests CI job. The script pulls the imported prompt with `include_model=True`, invokes it, and checks that the result matches the prompt's schema. The script passes `secrets={"OPENAI_API_KEY": }` to [`pull_prompt`](https://github.com/langchain-ai/langsmith-sdk/blob/47c553caf08c1989e2be3ff1886da3eb801f709f/python/langsmith/client.py#L10228), because the LangSmith Python SDK reads no secret from the environment unless asked and raises when a manifest holds a secret reference and no `secrets` map arrives. The script exists because the LangSmith API has no documented way to run a prompt with its model: the one prompt endpoint in the OpenAPI spec, `POST /api/v1/prompts/invoke_prompt`, takes message templates and inputs with no model or schema, and no LangSmith SDK calls it. In [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8), a developer calls that endpoint on the live API in case it does more than the spec says. The script needs an OpenAI API key. codekiln creates it and stores it in 1Password, in the repository's CI secrets and as the destination workspace's `OPENAI_API_KEY` secret, and in [export-import-prompts-and-assistants Task 2.1 Add the destination workspace and the model provider key](tasks.md#L12) a developer adds it to `fnox.toml` and the CI workflow. ### Export and import use the same SDK calls as the planned `prompt create`, `get` and `update` @@ -151,7 +151,7 @@ None. ### 3 - How should the end-to-end test show that an imported structured prompt still returns structured data? > Context from the drafting agent, for question 3. -> The LangSmith OpenAPI spec's only prompt-invoke endpoint, `POST /api/v1/prompts/invoke_prompt`, takes message templates and inputs with no model or schema, and no SDK calls it. In [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8), a developer calls it on the live API. The draft runs a Python script through a mise task, using the LangSmith Python SDK and LangChain, which is how most users run a prompt. It needs a model provider key, such as `OPENAI_API_KEY`, which codekiln would create and store in 1Password, the CI secrets and the destination workspace's secrets. Building the model call into langstar as `prompt verify --invoke` would need langstar to call each provider's API itself. Recommendation: the Python script, with an OpenAI key and a small model. +> The LangSmith OpenAPI spec's only prompt-invoke endpoint, `POST /api/v1/prompts/invoke_prompt`, takes message templates and inputs with no model or schema, and no SDK calls it. In [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8), a developer calls it on the live API. The draft runs a Python script through a mise task with `uv run --script`, using the LangSmith Python SDK and LangChain, which is how most users run a prompt; the script names both as inline dependencies, and the Integration Tests CI job installs `uv` and Python through `mise install`. It needs a model provider key, such as `OPENAI_API_KEY`, which codekiln would create and store in 1Password, the CI secrets and the destination workspace's secrets. Building the model call into langstar as `prompt verify --invoke` would need langstar to call each provider's API itself. Recommendation: the Python script, with an OpenAI key and a small model. diff --git a/openspec/changes/export-import-prompts-and-assistants/proposal.md b/openspec/changes/export-import-prompts-and-assistants/proposal.md index 4e44489bd..b34cb39c2 100644 --- a/openspec/changes/export-import-prompts-and-assistants/proposal.md +++ b/openspec/changes/export-import-prompts-and-assistants/proposal.md @@ -30,7 +30,7 @@ A prompt belongs to a workspace and an assistant belongs to a deployment, but bo ## Impact -- SDK: `sdk/src/prompts.rs` gains a commit read that asks for the model and returns it on `CommitManifestResponse`, a commit description on commit create, the new commit's `id`, the repo's readme, tags and `tenant_id` on `Prompt`, from which export fills `source.workspace_id`, a repo update, a commit list that returns each commit's `id`, calls to list, create and move commit tags, and a tag delete that `prompt import --update` and the end-to-end test use. `sdk/src/assistants.rs` gains the `context` and `version` fields, `assistant_id` and `if_exists` on create, `graph_id` on update, and a search by graph ID. `sdk/src/deployments.rs` gains the deployment's `url`. Each new public field breaks a crate outside this repository that builds that SDK struct with a struct literal or destructures it with a pattern that names every field; codekiln decides in [Should the SDK field additions count as a breaking change?](design.md#9---should-the-sdk-field-additions-count-as-a-breaking-change) whether that makes the release a major version. +- SDK: `sdk/src/prompts.rs` gains the prompt calls export and import need: a commit read that asks for the model and returns it on `CommitManifestResponse`, a commit create that sends a description and returns the new commit's `id`, a commit list, calls that list, create, move and delete commit tags, and a repo update. `Prompt` gains the repo's readme, tags and `tenant_id`, and export fills `source.workspace_id` from `tenant_id`. `sdk/src/assistants.rs` gains the `context` and `version` fields, `assistant_id` and `if_exists` on create, `graph_id` on update, and a search by graph ID. `sdk/src/deployments.rs` gains the deployment's `url`. Each new public field breaks a crate outside this repository that builds that SDK struct with a struct literal or destructures it with a pattern that names every field; codekiln decides in [Should the SDK field additions count as a breaking change?](design.md#9---should-the-sdk-field-additions-count-as-a-breaking-change) whether that makes the release a major version. - CLI: new subcommands in `cli/src/commands/prompt.rs` and `cli/src/commands/assistant.rs`, and a shared module for the export file, the destination record, comparison and the exit status. `resolve_deployment_url` in `cli/src/deployment_utils.rs` also returns the Agent Server URL of a deployment built from GitHub, which every `langstar assistant` command needs. - Tests need a second workspace, a second deployment and one model provider key for the structured output check. codekiln creates the workspace and the key, and langstar's tests read them through `fnox.toml` and the CI workflow; the destination deployment fixture from [export-import-prompts-and-assistants Task 2.4 Add a destination deployment fixture](tasks.md#L19) creates the deployment whenever the cleanup workflow has deleted it, as the source fixture does today. Nobody knows yet whether the control plane accepts a second deployment from this repository. In [export-import-prompts-and-assistants Task 2.3 Register the test graph under a second graph ID](tasks.md#L18), a developer deploys the new `langgraph.json` once to find out, before anyone builds the fixture. - Docs: a new usage page on moving prompts and assistants between environments. diff --git a/openspec/changes/export-import-prompts-and-assistants/tasks.md b/openspec/changes/export-import-prompts-and-assistants/tasks.md index ee5bdec64..bad381dcb 100644 --- a/openspec/changes/export-import-prompts-and-assistants/tasks.md +++ b/openspec/changes/export-import-prompts-and-assistants/tasks.md @@ -12,7 +12,7 @@ Each numbered group below is one pull request, merged in the order listed, excep - [ ] 2.1 Add the destination workspace and the model provider key to `fnox.toml`, the CI workflow and `docs/dev/environment-variables.md`, with placeholders only; verify with `mise run check` and a CI run that reads both. This task waits on codekiln to create a second workspace in the test organization and an OpenAI API key, and to store both in 1Password and the repository's CI secrets and the key as the destination workspace's `OPENAI_API_KEY` secret. The import tests need the workspace as a place to import into, and its secret lets prompt import's secret check pass for a prompt with an OpenAI model. The key lets the check in 7.3 run each imported structured prompt once against a small OpenAI model, which bills that account on every CI run; it is the one test that shows a copy still returns structured data, where every other test compares text. - [ ] 2.2 Make `resolve_deployment_url` in `cli/src/deployment_utils.rs` find the Agent Server URL of a deployment built from GitHub: - Add `url` to `Deployment` in `sdk/src/deployments.rs` as an `Option` with `#[serde(default)]`. The control-plane OpenAPI spec declares `url` nullable, and a plain `String` would make a deployment list fail to parse when one deployment in it has no URL yet. - - Add a method to `Deployment` that returns `url`, or `source_config.custom_url` when `url` is null, missing or empty, and have `resolve_deployment_url` call it. + - Add a method to `Deployment` that returns `url`, or `source_config.custom_url` when `url` is null, missing or empty, and have `resolve_deployment_url` call it. When the method returns neither, `resolve_deployment_url` returns an error saying that the deployment has no URL, in place of today's "has no custom_url in source_config", and a test asserts that message. - End the four `Deployment` struct literals in `sdk/tests/integration_deployment_workflow.rs` and `cli/src/commands/deployment.rs` with `..Default::default()`, so they still compile after the new field lands. - Verify with a unit test in `sdk/src/deployments.rs` for each case, among them a deployment whose `url` is null and one whose JSON has no `url` key, and by replacing the bare `#[ignore]` on `test_assistant_create_basic` in `cli/tests/assistant_command_test.rs` with `#[cfg_attr(not(feature = "integration-tests"), ignore)]`, the guard `test_assistant_description_field` beside it carries, and running it against the test deployment. - [ ] 2.3 Add a second `langgraph.json` under `tests/fixtures/` that registers the test graph from `tests/fixtures/test-graph-deployment/langgraph.json` under two graph IDs, `test_graph` and `test_graph_alt`, so the update tests in 3.3 and 10.2 can move an assistant from one graph to the other. To verify, deploy the new file while the source test deployment is live, which shows whether the control plane accepts a second deployment from this repository: run `langstar deployment create` with `--branch` set to this pull request's branch and `--config-path` set to the new file. A 409 "A deployment already exists for this agent environment" means it does not, and the developer reports that to codekiln before anyone starts task 2.4. Otherwise run `langstar assistant list --deployment `, find one assistant with graph ID `test_graph` and one with `test_graph_alt`, then delete the deployment. @@ -94,7 +94,7 @@ Each integration test in this group carries `#[cfg_attr(not(feature = "integrati - An integration test that `--update` against a destination whose latest commit matches the file and carries a `staging` tag the file lacks deletes `staging`, adds no commit and exits with status 0, and that `--dry-run` lists the delete. - An integration test that `--dry-run` writes nothing. - A unit test in `cli/src/commands/prompt.rs` that serves, through a `mockito` server and `LangchainClient::with_base_urls`, a commit read back that lacks a field the file holds, calls the import code, and asserts that the result names that field's JSON Pointer path and gives exit status 3, and that the destination record holds the new commit hash. -- [ ] 7.3 Add the structured output check as a mise task that pulls the imported prompt with `include_model=True` and a `secrets` map holding the model provider key read from the environment, invokes it and validates the result against the schema; verify by having the import integration test create a `json_schema` prompt and a `function_calling` prompt, each saved with a small OpenAI model, import both into the destination workspace, and run the check on each +- [ ] 7.3 Add the structured output check as a mise task that runs a Python script with `uv run --script`. The script names `langsmith` and `langchain-openai` as inline script dependencies, pulls the imported prompt with `include_model=True` and a `secrets` map holding the model provider key read from the environment, invokes it and validates the result against the schema. Add `uv` to `mise.toml`, and remove `install: false` from the Setup mise step of the Integration Tests job in `.github/workflows/ci.yml` so that job installs `uv` and Python 3.11. Verify by having the import integration test create a `json_schema` prompt and a `function_calling` prompt, each saved with a small OpenAI model, import both into the destination workspace and run the check on each, under `mise run check` and in that CI job. - [ ] 7.4 Document prompt import, the workspace scope of prompts and setting up destination credentials; verify that each documented command runs as written ## 8. assistant export From 59b528d3ae46b165effcfb5e85f0ade00f9a3b52 Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Wed, 14 Oct 2026 02:23:35 -0400 Subject: [PATCH 53/90] =?UTF-8?q?=F0=9F=93=9A=20docs(openspec):=20drop=20t?= =?UTF-8?q?he=20deployment=20url=20change,=20since=20GitHub=20deployments?= =?UTF-8?q?=20already=20carry=20custom=5Furl,=20and=20split=20task=201.2?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The control-plane OpenAPI spec says SourceConfig.custom_url is set automatically for github source deployments, and test_assistant_description_field already passes against the GitHub-built test deployment. Task 2.2 now only re-enables test_assistant_create_basic. Also fix the question 3 context about the CI job's mise install, and split task 1.2 into sub-bullets. #807 Export, import, compare, and verify prompts and assistants across LangSmith environments Co-Authored-By: Claude Opus 5.5 --- .../design.md | 14 ++++------ .../proposal.md | 6 ++-- .../specs/assistant/spec.md | 28 ------------------- .../tasks.md | 18 ++++++------ 4 files changed, 17 insertions(+), 49 deletions(-) diff --git a/openspec/changes/export-import-prompts-and-assistants/design.md b/openspec/changes/export-import-prompts-and-assistants/design.md index cf3d2b141..63fa0f8fc 100644 --- a/openspec/changes/export-import-prompts-and-assistants/design.md +++ b/openspec/changes/export-import-prompts-and-assistants/design.md @@ -4,7 +4,7 @@ `langstar prompt push` in `cli/src/commands/prompt.rs` builds one shape of prompt: a single human message, with an optional schema from [#402 structured output prompts](https://github.com/codekiln/langstar/issues/402). `langstar prompt pull` prints any manifest LangSmith returns, but `PromptClient` in `sdk/src/prompts.rs` never asks LangSmith for the model, so that manifest has none. `PromptClient` keeps the manifest as raw JSON and has a typed `StructuredPrompt` view that holds messages, the schema and the method. -`langstar assistant` in `cli/src/commands/assistant.rs` takes `--deployment` on every subcommand and sends its requests to the Agent Server URL that `resolve_deployment_url` in `cli/src/deployment_utils.rs` reads from the deployment's `source_config.custom_url`. A deployment built from GitHub has no `custom_url`, so every `langstar assistant` command fails on one today. `AssistantClient` in `sdk/src/assistants.rs` has no `context` or `version` field and its search cannot filter by graph ID. +`langstar assistant` in `cli/src/commands/assistant.rs` takes `--deployment` on every subcommand and sends its requests to the Agent Server URL that `resolve_deployment_url` in `cli/src/deployment_utils.rs` reads from the deployment's `source_config.custom_url`. The control plane fills in `source_config.custom_url` for a deployment built from GitHub, as `SourceConfig.custom_url` in `reference/openapi/langchain/control-plane/openapi.json` says, so `langstar assistant` already reaches the test deployment: `test_assistant_description_field` in `cli/tests/assistant_command_test.rs` creates and reads an assistant there on every CI run. `AssistantClient` in `sdk/src/assistants.rs` has no `context` or `version` field and its search cannot filter by graph ID. Langstar reads one API key and one workspace at a time, from flags, environment variables or the config file. Langstar exits with status 1 when a command fails, and clap, the library that parses langstar's flags, exits with status 2 when it rejects a mistyped flag or a missing required one. @@ -30,10 +30,6 @@ Export runs with the source environment's credentials and writes a file. Import The LangSmith Go CLI moves a prompt through a file the same way, as `references/langsmith-cli.md` shows. Langstar's export asks for the model, and the Go CLI's pull does not. Langstar's import reads the file export writes, while the Go CLI's push sends whatever JSON it reads as the manifest, so its own pull output reaches LangSmith wrapped in an envelope. Langstar's import also requires `--workspace-id`, so langstar imports into the workspace the user typed on the command line, even when `LANGSMITH_WORKSPACE_ID` names another. -### Assistant commands send their requests to the deployment's `url` - -The control plane returns each deployment's Agent Server address in a top-level `url`, which the SDK's `Deployment` drops today. In [export-import-prompts-and-assistants Task 2.2 Return the URL of a deployment built from GitHub](tasks.md#L13), a developer adds `url` to `Deployment` and has `resolve_deployment_url` use it, falling back to `source_config.custom_url` when `url` is null, missing or empty. Every `langstar assistant` command then reaches a deployment built from GitHub, and the assistant export, compare, verify and import commands are built after that change merges. - ### The export file keeps the manifest LangSmith returned Export stores the manifest as raw JSON. The typed `StructuredPrompt` struct cannot hold a model binding, placeholders, several message types or partial variables, so routing through it would drop them. Langstar reads the manifest through typed views only to fill in `requires`. @@ -111,7 +107,7 @@ Status 0 means the resource matches and every check passed. Status 3 means langs ### A Python script checks that an imported structured prompt still returns structured data -A mise task runs a short Python script with `uv run --script`, which installs the `langsmith` and `langchain-openai` packages the script names as inline dependencies, locally and in the Integration Tests CI job. The script pulls the imported prompt with `include_model=True`, invokes it, and checks that the result matches the prompt's schema. The script passes `secrets={"OPENAI_API_KEY": }` to [`pull_prompt`](https://github.com/langchain-ai/langsmith-sdk/blob/47c553caf08c1989e2be3ff1886da3eb801f709f/python/langsmith/client.py#L10228), because the LangSmith Python SDK reads no secret from the environment unless asked and raises when a manifest holds a secret reference and no `secrets` map arrives. The script exists because the LangSmith API has no documented way to run a prompt with its model: the one prompt endpoint in the OpenAPI spec, `POST /api/v1/prompts/invoke_prompt`, takes message templates and inputs with no model or schema, and no LangSmith SDK calls it. In [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8), a developer calls that endpoint on the live API in case it does more than the spec says. The script needs an OpenAI API key. codekiln creates it and stores it in 1Password, in the repository's CI secrets and as the destination workspace's `OPENAI_API_KEY` secret, and in [export-import-prompts-and-assistants Task 2.1 Add the destination workspace and the model provider key](tasks.md#L12) a developer adds it to `fnox.toml` and the CI workflow. +A mise task runs a short Python script with `uv run --script`, which installs the `langsmith` and `langchain-openai` packages the script names as inline dependencies, locally and in the Integration Tests CI job. The script pulls the imported prompt with `include_model=True`, invokes it, and checks that the result matches the prompt's schema. The script passes `secrets={"OPENAI_API_KEY": }` to [`pull_prompt`](https://github.com/langchain-ai/langsmith-sdk/blob/47c553caf08c1989e2be3ff1886da3eb801f709f/python/langsmith/client.py#L10228), because the LangSmith Python SDK reads no secret from the environment unless asked and raises when a manifest holds a secret reference and no `secrets` map arrives. The script exists because the LangSmith API has no documented way to run a prompt with its model: the one prompt endpoint in the OpenAPI spec, `POST /api/v1/prompts/invoke_prompt`, takes message templates and inputs with no model or schema, and no LangSmith SDK calls it. In [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8), a developer calls that endpoint on the live API in case it does more than the spec says. The script needs an OpenAI API key. codekiln creates it and stores it in 1Password, in the repository's CI secrets and as the destination workspace's `OPENAI_API_KEY` secret, and in [export-import-prompts-and-assistants Task 2.1 Add the destination workspace and the model provider key](tasks.md#L16) a developer adds it to `fnox.toml` and the CI workflow. ### Export and import use the same SDK calls as the planned `prompt create`, `get` and `update` @@ -121,7 +117,7 @@ A mise task runs a short Python script with `uv run --script`, which installs th - [LangSmith may store a pushed model differently from the source] → Import reads the prompt back and reports any difference, and in [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8) a developer checks the round trip against the live API before anyone builds the commands. - [Pushing a manifest identical to the latest commit may return 409] → Import compares first and pushes nothing when the prompt matches. -- [The SDK changes for export and import add public fields to `Deployment`, `Prompt`, `CommitManifestResponse`, `CommitData`, `CommitRequest`, `Assistant` and the assistant request structs, so a crate that builds one of those structs with a struct literal or destructures it with a pattern that names every field stops compiling] → Each pull request that adds a field updates every struct literal of that struct in this repository. [Should the SDK field additions count as a breaking change?](#9---should-the-sdk-field-additions-count-as-a-breaking-change) asks codekiln whether those pull requests also carry a `BREAKING CHANGE:` footer, which `docs/dev/git-scm-conventions.md` says makes the release a major version. +- [The SDK changes for export and import add public fields to `Prompt`, `CommitManifestResponse`, `CommitData`, `CommitRequest`, `Assistant` and the assistant request structs, so a crate that builds one of those structs with a struct literal or destructures it with a pattern that names every field stops compiling] → Each pull request that adds a field updates every struct literal of that struct in this repository. [Should the SDK field additions count as a breaking change?](#9---should-the-sdk-field-additions-count-as-a-breaking-change) asks codekiln whether those pull requests also carry a `BREAKING CHANGE:` footer, which `docs/dev/git-scm-conventions.md` says makes the release a major version. - [A prompt's manifest may hold a credential written as a literal model setting, rather than as a secret reference] → Export refuses with status 1 and writes nothing when a chat model setting whose name ends in `api_key` holds a string, and names that setting's path without its value. A credential under any other name, such as a header, still reaches the file, because langstar cannot tell it from any other string. The prompt export docs in [export-import-prompts-and-assistants Task 5.2 Document prompt export](tasks.md#L56) tell users to keep credentials in the workspace's secrets and to read an export file before committing it. - [An assistant's configuration, context or metadata may hold a credential, which export copies into a file meant for Git] → Export writes those objects as the Agent Server returns them, since a credential there looks like any other string. The assistant export docs in [export-import-prompts-and-assistants Task 8.2 Document assistant export](tasks.md#L109) tell users to keep credentials in the deployment's environment variables or the workspace's secrets, and to read an export file before committing it. - [A tag in the destination may already point at another commit] → `--update` moves the tag, and `--dry-run` lists the move. @@ -151,7 +147,7 @@ None. ### 3 - How should the end-to-end test show that an imported structured prompt still returns structured data? > Context from the drafting agent, for question 3. -> The LangSmith OpenAPI spec's only prompt-invoke endpoint, `POST /api/v1/prompts/invoke_prompt`, takes message templates and inputs with no model or schema, and no SDK calls it. In [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8), a developer calls it on the live API. The draft runs a Python script through a mise task with `uv run --script`, using the LangSmith Python SDK and LangChain, which is how most users run a prompt; the script names both as inline dependencies, and the Integration Tests CI job installs `uv` and Python through `mise install`. It needs a model provider key, such as `OPENAI_API_KEY`, which codekiln would create and store in 1Password, the CI secrets and the destination workspace's secrets. Building the model call into langstar as `prompt verify --invoke` would need langstar to call each provider's API itself. Recommendation: the Python script, with an OpenAI key and a small model. +> The LangSmith OpenAPI spec's only prompt-invoke endpoint, `POST /api/v1/prompts/invoke_prompt`, takes message templates and inputs with no model or schema, and no SDK calls it. In [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8), a developer calls it on the live API. The draft runs a Python script through a mise task with `uv run --script`, using the LangSmith Python SDK and LangChain, which is how most users run a prompt; the script names both as inline dependencies, and [export-import-prompts-and-assistants Task 7.3 Add the structured output check](tasks.md#L97) adds `uv` to `mise.toml` and removes `install: false` from the Integration Tests CI job, so that job runs `mise install`. It needs a model provider key, such as `OPENAI_API_KEY`, which codekiln would create and store in 1Password, the CI secrets and the destination workspace's secrets. Building the model call into langstar as `prompt verify --invoke` would need langstar to call each provider's API itself. Recommendation: the Python script, with an OpenAI key and a small model. @@ -193,6 +189,6 @@ None. ### 9 - Should the SDK field additions count as a breaking change? > Context from the drafting agent, for question 9. -> A new public field breaks only a crate that builds the struct with a struct literal or destructures it with a pattern that names every field; nothing in this repository destructures these structs that way. The only crate known to use the SDK is langstar's own CLI, and each pull request that adds a field updates the CLI's struct literals. `sdk/Cargo.toml` leaves the crate publishable, though, so an outside crate could break, and `docs/dev/git-scm-conventions.md` gives every API break a `BREAKING CHANGE:` footer, which makes the release a major version. The structs that gain fields are `Deployment`, `Prompt` (which `PromptClient::get` returns), `CommitManifestResponse` (which `PromptClient::get_commit` returns), `CommitData`, `CommitRequest`, `Assistant` and the assistant request structs. Marking them `#[non_exhaustive]` would stop every crate except the SDK itself from building them with struct literals, even with `..Default::default()`. langstar's own CLI builds `Assistant`, `CommitRequest`, `CreateAssistantRequest`, `UpdateAssistantRequest` and `Deployment` with struct literals today, and the SDK's integration tests build all of them except `Assistant` the same way, so the SDK would need a constructor for each of those five structs before the CLI and those tests compile again. After that one break, a new field would break nothing. [🚨 BREAKING CHANGE(sdk): move annotation-queue run calls to the platform items API · Pull Request #815](https://github.com/codekiln/langstar/pull/815) removes SDK functions and changes return types, marks itself breaking, and leaves that choice to codekiln too, so one answer to this question can set the policy for both pull requests. Recommendation: treat the additions as `feat` changes with no `BREAKING CHANGE:` footer, and name the new fields in the release notes. +> A new public field breaks only a crate that builds the struct with a struct literal or destructures it with a pattern that names every field; nothing in this repository destructures these structs that way. The only crate known to use the SDK is langstar's own CLI, and each pull request that adds a field updates the CLI's struct literals. `sdk/Cargo.toml` leaves the crate publishable, though, so an outside crate could break, and `docs/dev/git-scm-conventions.md` gives every API break a `BREAKING CHANGE:` footer, which makes the release a major version. The structs that gain fields are `Prompt` (which `PromptClient::get` returns), `CommitManifestResponse` (which `PromptClient::get_commit` returns), `CommitData`, `CommitRequest`, `Assistant` and the assistant request structs. Marking them `#[non_exhaustive]` would stop every crate except the SDK itself from building them with struct literals, even with `..Default::default()`. langstar's own CLI builds `Assistant`, `CommitRequest`, `CreateAssistantRequest` and `UpdateAssistantRequest` with struct literals today, and the SDK's integration tests build all of them except `Assistant` the same way, so the SDK would need a constructor for each of those four structs before the CLI and those tests compile again. After that one break, a new field would break nothing. [🚨 BREAKING CHANGE(sdk): move annotation-queue run calls to the platform items API · Pull Request #815](https://github.com/codekiln/langstar/pull/815) removes SDK functions and changes return types, marks itself breaking, and leaves that choice to codekiln too, so one answer to this question can set the policy for both pull requests. Recommendation: treat the additions as `feat` changes with no `BREAKING CHANGE:` footer, and name the new fields in the release notes. diff --git a/openspec/changes/export-import-prompts-and-assistants/proposal.md b/openspec/changes/export-import-prompts-and-assistants/proposal.md index b34cb39c2..770d3094b 100644 --- a/openspec/changes/export-import-prompts-and-assistants/proposal.md +++ b/openspec/changes/export-import-prompts-and-assistants/proposal.md @@ -26,12 +26,12 @@ A prompt belongs to a workspace and an assistant belongs to a deployment, but bo ### Modified Capabilities -- `assistant`: adds export, import, compare and verify requirements to `langstar assistant`, and lets every assistant command reach a deployment built from GitHub through the deployment's `url`. +- `assistant`: adds export, import, compare and verify requirements to `langstar assistant`. ## Impact -- SDK: `sdk/src/prompts.rs` gains the prompt calls export and import need: a commit read that asks for the model and returns it on `CommitManifestResponse`, a commit create that sends a description and returns the new commit's `id`, a commit list, calls that list, create, move and delete commit tags, and a repo update. `Prompt` gains the repo's readme, tags and `tenant_id`, and export fills `source.workspace_id` from `tenant_id`. `sdk/src/assistants.rs` gains the `context` and `version` fields, `assistant_id` and `if_exists` on create, `graph_id` on update, and a search by graph ID. `sdk/src/deployments.rs` gains the deployment's `url`. Each new public field breaks a crate outside this repository that builds that SDK struct with a struct literal or destructures it with a pattern that names every field; codekiln decides in [Should the SDK field additions count as a breaking change?](design.md#9---should-the-sdk-field-additions-count-as-a-breaking-change) whether that makes the release a major version. -- CLI: new subcommands in `cli/src/commands/prompt.rs` and `cli/src/commands/assistant.rs`, and a shared module for the export file, the destination record, comparison and the exit status. `resolve_deployment_url` in `cli/src/deployment_utils.rs` also returns the Agent Server URL of a deployment built from GitHub, which every `langstar assistant` command needs. +- SDK: `sdk/src/prompts.rs` gains the prompt calls export and import need: a commit read that asks for the model and returns it on `CommitManifestResponse`, a commit create that sends a description and returns the new commit's `id`, a commit list, calls that list, create, move and delete commit tags, and a repo update. `Prompt` gains the repo's readme, tags and `tenant_id`, and export fills `source.workspace_id` from `tenant_id`. `sdk/src/assistants.rs` gains the `context` and `version` fields, `assistant_id` and `if_exists` on create, `graph_id` on update, and a search by graph ID. Each new public field breaks a crate outside this repository that builds that SDK struct with a struct literal or destructures it with a pattern that names every field; codekiln decides in [Should the SDK field additions count as a breaking change?](design.md#9---should-the-sdk-field-additions-count-as-a-breaking-change) whether that makes the release a major version. +- CLI: new subcommands in `cli/src/commands/prompt.rs` and `cli/src/commands/assistant.rs`, and a shared module for the export file, the destination record, comparison and the exit status. - Tests need a second workspace, a second deployment and one model provider key for the structured output check. codekiln creates the workspace and the key, and langstar's tests read them through `fnox.toml` and the CI workflow; the destination deployment fixture from [export-import-prompts-and-assistants Task 2.4 Add a destination deployment fixture](tasks.md#L19) creates the deployment whenever the cleanup workflow has deleted it, as the source fixture does today. Nobody knows yet whether the control plane accepts a second deployment from this repository. In [export-import-prompts-and-assistants Task 2.3 Register the test graph under a second graph ID](tasks.md#L18), a developer deploys the new `langgraph.json` once to find out, before anyone builds the fixture. - Docs: a new usage page on moving prompts and assistants between environments. diff --git a/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md b/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md index b6b9cf907..c957b7f46 100644 --- a/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md +++ b/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md @@ -1,31 +1,3 @@ -## MODIFIED Requirements - -### Requirement: Every assistant command targets one deployment - -Every `langstar assistant` subcommand SHALL take a required `--deployment` flag holding a deployment name or ID. Langstar SHALL look the deployment up through the control plane API and send the assistant request to the deployment's `url`, or to its `source_config.custom_url` when `url` is null, missing or empty. - -#### Scenario: Deployment found by name - -- **WHEN** a user runs `langstar assistant list --deployment my-agent` -- **AND** the workspace has a deployment named `my-agent` -- **THEN** langstar lists the assistants on that deployment - -#### Scenario: Deployment built from GitHub - -- **WHEN** a user runs `langstar assistant list --deployment my-agent` -- **AND** `my-agent` was built from a GitHub repository and has a `url` but no `source_config.custom_url` -- **THEN** langstar lists the assistants on that deployment - -#### Scenario: Deployment not found - -- **WHEN** a user passes a `--deployment` that matches no deployment name or ID in the workspace -- **THEN** langstar exits with an error that names the deployment and suggests `langstar deployment list` - -#### Scenario: Deployment has no URL - -- **WHEN** the deployment exists but has neither a `url` nor a custom URL -- **THEN** langstar exits with an error saying the deployment has no URL - ## ADDED Requirements ### Requirement: Export an assistant to a file diff --git a/openspec/changes/export-import-prompts-and-assistants/tasks.md b/openspec/changes/export-import-prompts-and-assistants/tasks.md index bad381dcb..3e1474648 100644 --- a/openspec/changes/export-import-prompts-and-assistants/tasks.md +++ b/openspec/changes/export-import-prompts-and-assistants/tasks.md @@ -1,26 +1,26 @@ # Tasks -Each numbered group below is one pull request, merged in the order listed, except group 2, whose tasks each get a pull request of their own. Tasks 2.2, 2.3 and 2.4 merge in that order: the check in 2.3 lists assistants through the deployment URL that 2.2 resolves, and the fixture in 2.4 deploys from `main`, so the `langgraph.json` from 2.3 has to be on `main` first. Task 2.1 waits on codekiln to create the destination workspace and an OpenAI key, and can merge any time before group 6, whose prompt compare and verify tests are the first to use that workspace. +Each numbered group below is one pull request, merged in the order listed, except group 2, whose tasks each get a pull request of their own. Tasks 2.3 and 2.4 merge in that order: the fixture in 2.4 deploys from `main`, so the `langgraph.json` from 2.3 has to be on `main` first. Task 2.2 can merge at any time. Task 2.1 waits on codekiln to create the destination workspace and an OpenAI key, and can merge any time before group 6, whose prompt compare and verify tests are the first to use that workspace. ## 1. Check the API round trip before building on it - [ ] 1.1 Rerun the jq query for `prompt-endpoints.json` in `reference/api-specs/langsmith/FRAGMENTS.md`, so the extract picks up the `include` parameter of the commit read that its 2025-11-29 copy lacks, and extract the workspace secrets endpoints (`/api/v1/workspaces/current/secrets`) from `reference/openapi/langchain/langsmith/openapi.json` into a new file in `reference/api-specs/langsmith/` with its jq query in `FRAGMENTS.md`; verify by rerunning each query and getting the same file -- [ ] 1.2 Write `reference/research/807-openapi-validation.md`, recording from the live API: the manifest returned with `include=model` for a structured prompt with a model, the outer `id` of that manifest, the `model_config` and `model_provider` values returned beside it, whether pushing that manifest to a second workspace returns the same manifest, `model_config` and `model_provider`, whether `model_provider` matches the class name export writes to `requires.model_class`, what pushing an identical manifest returns, what the Agent Server returns for `context`, whether an Agent Server assistant update that sends a metadata key with a null value removes that key, whether an update that sends `config` or `context` replaces the assistant's object or merges into it, whether an update can clear an assistant's description, and what `POST /api/v1/prompts/invoke_prompt` returns for a prompt with a model; verify that every claim in the report names the request that produced it +- [ ] 1.2 Write `reference/research/807-openapi-validation.md`, recording what the live API returns. Verify that every claim in the report names the request that produced it. + - For a structured prompt saved with a model, read with `include=model`: the manifest, its outer `id`, the `model_config` and `model_provider` beside it, and whether `model_provider` matches the class name export writes to `requires.model_class`. + - Push that manifest to a second workspace: whether the read-back returns the same manifest, `model_config` and `model_provider`, and what pushing an identical manifest again returns. + - On the Agent Server: what it returns for `context`, whether an assistant update that sends a metadata key with a null value removes that key, whether an update that sends `config` or `context` replaces the assistant's object or merges into it, and whether an update can clear the description. + - What `POST /api/v1/prompts/invoke_prompt` returns for a prompt with a model. ## 2. Test environments - [ ] 2.1 Add the destination workspace and the model provider key to `fnox.toml`, the CI workflow and `docs/dev/environment-variables.md`, with placeholders only; verify with `mise run check` and a CI run that reads both. This task waits on codekiln to create a second workspace in the test organization and an OpenAI API key, and to store both in 1Password and the repository's CI secrets and the key as the destination workspace's `OPENAI_API_KEY` secret. The import tests need the workspace as a place to import into, and its secret lets prompt import's secret check pass for a prompt with an OpenAI model. The key lets the check in 7.3 run each imported structured prompt once against a small OpenAI model, which bills that account on every CI run; it is the one test that shows a copy still returns structured data, where every other test compares text. -- [ ] 2.2 Make `resolve_deployment_url` in `cli/src/deployment_utils.rs` find the Agent Server URL of a deployment built from GitHub: - - Add `url` to `Deployment` in `sdk/src/deployments.rs` as an `Option` with `#[serde(default)]`. The control-plane OpenAPI spec declares `url` nullable, and a plain `String` would make a deployment list fail to parse when one deployment in it has no URL yet. - - Add a method to `Deployment` that returns `url`, or `source_config.custom_url` when `url` is null, missing or empty, and have `resolve_deployment_url` call it. When the method returns neither, `resolve_deployment_url` returns an error saying that the deployment has no URL, in place of today's "has no custom_url in source_config", and a test asserts that message. - - End the four `Deployment` struct literals in `sdk/tests/integration_deployment_workflow.rs` and `cli/src/commands/deployment.rs` with `..Default::default()`, so they still compile after the new field lands. - - Verify with a unit test in `sdk/src/deployments.rs` for each case, among them a deployment whose `url` is null and one whose JSON has no `url` key, and by replacing the bare `#[ignore]` on `test_assistant_create_basic` in `cli/tests/assistant_command_test.rs` with `#[cfg_attr(not(feature = "integration-tests"), ignore)]`, the guard `test_assistant_description_field` beside it carries, and running it against the test deployment. +- [ ] 2.2 Replace the bare `#[ignore]` on `test_assistant_create_basic` in `cli/tests/assistant_command_test.rs`, whose reason says a deployment built from GitHub has no `custom_url`, with `#[cfg_attr(not(feature = "integration-tests"), ignore)]`, the guard `test_assistant_description_field` beside it carries. The control plane sets `source_config.custom_url` on a deployment built from GitHub, as `SourceConfig.custom_url` in `reference/openapi/langchain/control-plane/openapi.json` says. Verify by running the test against the test deployment. - [ ] 2.3 Add a second `langgraph.json` under `tests/fixtures/` that registers the test graph from `tests/fixtures/test-graph-deployment/langgraph.json` under two graph IDs, `test_graph` and `test_graph_alt`, so the update tests in 3.3 and 10.2 can move an assistant from one graph to the other. To verify, deploy the new file while the source test deployment is live, which shows whether the control plane accepts a second deployment from this repository: run `langstar deployment create` with `--branch` set to this pull request's branch and `--config-path` set to the new file. A 409 "A deployment already exists for this agent environment" means it does not, and the developer reports that to codekiln before anyone starts task 2.4. Otherwise run `langstar assistant list --deployment `, find one assistant with graph ID `test_graph` and one with `test_graph_alt`, then delete the deployment. -- [ ] 2.4 Add a destination deployment fixture to `sdk/src/test_utils.rs` and `cli/tests/common/fixtures.rs` that deploys the `langgraph.json` from 2.3 under the name prefix `dest-integration-test`. `find_reusable_deployment` returns any deployment whose name starts with the fixture's prefix before it checks the graph config, so this prefix keeps the source fixture, whose prefix is `pr-integration-test`, from picking up the destination deployment. `find_reusable_deployment` then falls back to any live deployment built from the same repository and graph config file, so the separate file from 2.3 keeps the destination fixture from picking up the source deployment. The control plane answers a second deployment of one agent environment with 409 "A deployment already exists for this agent environment". No document says which fields make an agent environment, and the verification in 2.3 shows whether a second graph config file in the same repository makes a new one. `.github/workflows/cleanup-test-deployments.yml` runs every four hours and deletes each deployment whose name contains `integration-test` once it is four hours old or older, so the destination deployment is deleted and rebuilt on the same schedule as the source deployment. Verify with an integration test that asserts the destination and source deployment IDs differ and lists assistants on the destination deployment through the URL that 2.2 resolves +- [ ] 2.4 Add a destination deployment fixture to `sdk/src/test_utils.rs` and `cli/tests/common/fixtures.rs` that deploys the `langgraph.json` from 2.3 under the name prefix `dest-integration-test`. `find_reusable_deployment` returns any deployment whose name starts with the fixture's prefix before it checks the graph config, so this prefix keeps the source fixture, whose prefix is `pr-integration-test`, from picking up the destination deployment. `find_reusable_deployment` then falls back to any live deployment built from the same repository and graph config file, so the separate file from 2.3 keeps the destination fixture from picking up the source deployment. The control plane answers a second deployment of one agent environment with 409 "A deployment already exists for this agent environment". No document says which fields make an agent environment, and the verification in 2.3 shows whether a second graph config file in the same repository makes a new one. `.github/workflows/cleanup-test-deployments.yml` runs every four hours and deletes each deployment whose name contains `integration-test` once it is four hours old or older, so the destination deployment is deleted and rebuilt on the same schedule as the source deployment. Verify with an integration test that asserts the destination and source deployment IDs differ and lists assistants on the destination deployment with `langstar assistant list` ## 3. SDK reads and writes the fields export and import need -Each task in this group adds public fields to SDK structs and updates every struct literal of those structs in this repository, so the workspace still compiles. The structs are `Prompt`, `CommitManifestResponse`, `CommitData`, `CommitRequest`, `Assistant`, `CreateAssistantRequest`, `UpdateAssistantRequest` and `AssistantSearchRequest`. A crate outside this repository that builds one of them with a struct literal stops compiling too, and codekiln decides in [Should the SDK field additions count as a breaking change?](design.md#9---should-the-sdk-field-additions-count-as-a-breaking-change) whether this group's pull request and task 2.2's carry a `BREAKING CHANGE:` footer. +Each task in this group adds public fields to SDK structs and updates every struct literal of those structs in this repository, so the workspace still compiles. The structs are `Prompt`, `CommitManifestResponse`, `CommitData`, `CommitRequest`, `Assistant`, `CreateAssistantRequest`, `UpdateAssistantRequest` and `AssistantSearchRequest`. A crate outside this repository that builds one of them with a struct literal stops compiling too, and codekiln decides in [Should the SDK field additions count as a breaking change?](design.md#9---should-the-sdk-field-additions-count-as-a-breaking-change) whether this group's pull request carries a `BREAKING CHANGE:` footer. Each integration test in this group carries `#[cfg_attr(not(feature = "integration-tests"), ignore)]`, as the tests in `sdk/tests/structured_prompts_integration_test.rs` do, so that the SDK Integration Tests job in `.github/workflows/ci.yml` and `mise run check` both run it. From ee468954e960d99354aed2233a39767390334d02 Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Wed, 14 Oct 2026 02:31:08 -0400 Subject: [PATCH 54/90] =?UTF-8?q?=F0=9F=93=9A=20docs(openspec):=20give=20a?= =?UTF-8?q?ssistant=20compare=20and=20verify=20--record,=20sort=20repo=20t?= =?UTF-8?q?ags=20and=20secrets=20in=20the=20export,=20and=20test=20JSON=20?= =?UTF-8?q?Pointer=20escaping?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit #807 Export, import, compare, and verify prompts and assistants across LangSmith environments Co-Authored-By: Claude Opus 5.5 --- .../export-import-prompts-and-assistants/design.md | 2 +- .../specs/assistant/spec.md | 8 +++++++- .../specs/prompt/spec.md | 2 +- .../export-import-prompts-and-assistants/tasks.md | 12 ++++++------ 4 files changed, 15 insertions(+), 9 deletions(-) diff --git a/openspec/changes/export-import-prompts-and-assistants/design.md b/openspec/changes/export-import-prompts-and-assistants/design.md index 63fa0f8fc..9bdd70554 100644 --- a/openspec/changes/export-import-prompts-and-assistants/design.md +++ b/openspec/changes/export-import-prompts-and-assistants/design.md @@ -65,7 +65,7 @@ Import writes the IDs of what it created to `.destinations.json`, key A repeated prompt import finds the copy by its handle and uses the record to tell whether someone changed it. Import writes the new commit hash to the record before it reads the commit back, so a read-back that finds a difference still leaves the hash in the record. The next import compares the destination's latest commit with that hash, and reports a change only when someone else pushed a commit after langstar's. Langstar can stop after it pushes a commit and before it writes that commit's hash to the record. The next import finds the destination's latest commit matching the file, writes its hash to the record and reports the prompt unchanged. -An assistant's ID differs on every deployment, so the record is the only way a repeated import finds the assistant it made. Import chooses the new assistant's ID itself and writes it to the record, marked pending, before it sends the create request with that `assistant_id` and `if_exists` set to `do_nothing`, which the Agent Server's `AssistantCreate` accepts. After the create succeeds, import marks the entry created. If langstar stops before then, the next import finds the pending entry and sends the same create again, and the Agent Server returns the assistant the first import made. +An assistant's ID differs on every deployment, so the record is the only way a repeated import finds the assistant it made. Import chooses the new assistant's ID itself and writes it to the record, marked pending, before it sends the create request with that `assistant_id` and `if_exists` set to `do_nothing`, which the Agent Server's `AssistantCreate` accepts. After the create succeeds, import marks the entry created. If langstar stops before then, the next import finds the pending entry and sends the same create again, and the Agent Server returns the assistant the first import made. Assistant compare and verify find the assistant through the same record, and take `--record ` as import does, so a record kept at another path still works. ### Import changes an existing resource only with --update diff --git a/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md b/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md index c957b7f46..d600ca447 100644 --- a/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md +++ b/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md @@ -150,7 +150,7 @@ After writing, `langstar assistant import` SHALL record the assistant ID in the ### Requirement: Compare an assistant with its export file -`langstar assistant compare --deployment ` SHALL compare the file with the recorded assistant, or the one named by `--assistant-id`, and list each difference by its JSON path with both values. It SHALL compare only the fields the export file holds, leaving out the assistant ID, version and timestamps the server sets. +`langstar assistant compare --deployment ` SHALL compare the file with the assistant recorded in the destination record (`.destinations.json` unless `--record ` names another), or the one named by `--assistant-id`, and list each difference by its JSON path with both values. It SHALL compare only the fields the export file holds, leaving out the assistant ID, version and timestamps the server sets. #### Scenario: Metadata changed on the deployment @@ -163,6 +163,12 @@ After writing, `langstar assistant import` SHALL record the assistant ID in the - **AND** a user runs `langstar assistant compare support-bot.json --deployment my-agent-prod` without `--assistant-id` - **THEN** langstar exits with status 1 and an error that names the deployment and suggests `--assistant-id` +#### Scenario: Record kept at another path + +- **WHEN** a user imported `support-bot.json` with `--record records/prod.json` +- **AND** runs `langstar assistant compare support-bot.json --deployment my-agent-prod --record records/prod.json` +- **THEN** langstar compares the file with the assistant that import recorded in `records/prod.json` + ### Requirement: Verify that an assistant can run on a deployment `langstar assistant verify --deployment ` SHALL do what `compare` does and also check that the deployment serves the file's graph ID and that the workspace, from `--workspace-id` or the configured workspace, has each prompt under `requires.prompts`. It SHALL name each missing graph and prompt and exit with status 3 when any check fails. diff --git a/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md index c2409c645..3acbbd4b8 100644 --- a/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md +++ b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md @@ -42,7 +42,7 @@ ### Requirement: Write the same file for the same commit -Exporting the same commit twice SHALL produce identical files when nobody changed the repo details or the tags that point at the commit in between, since those change without a new commit. The file SHALL hold no export time, and langstar SHALL write object keys in a fixed order and the commit's tags sorted by name. +Exporting the same commit twice SHALL produce identical files when nobody changed the repo details or the tags that point at the commit in between, since those change without a new commit. The file SHALL hold no export time, and langstar SHALL write object keys in a fixed order and sort `commit.tags`, `repo.tags` and `requires.secrets` by name. Compare SHALL sort the destination's tags the same way before comparing. #### Scenario: Export twice diff --git a/openspec/changes/export-import-prompts-and-assistants/tasks.md b/openspec/changes/export-import-prompts-and-assistants/tasks.md index 3e1474648..402d49219 100644 --- a/openspec/changes/export-import-prompts-and-assistants/tasks.md +++ b/openspec/changes/export-import-prompts-and-assistants/tasks.md @@ -31,13 +31,13 @@ Each integration test in this group carries `#[cfg_attr(not(feature = "integrati ## 4. Shared code for the prompt and assistant commands: the export file, the comparison and the report -- [ ] 4.1 Add the export file types, with sorted keys, no export time and a reader that accepts only `format_version` 1 and the expected `kind`, and the destination record file, which langstar replaces by writing a temporary file in the same directory and renaming it over the old one, to a new module in `cli/src/`. Verify with these unit tests: - - Tests that build equivalent nested values with keys inserted in different orders, serialize each, and assert identical text with keys sorted at every level. +- [ ] 4.1 Add the export file types, with sorted keys, sorted `commit.tags`, `repo.tags` and `requires.secrets`, no export time and a reader that accepts only `format_version` 1 and the expected `kind`, and the destination record file, which langstar replaces by writing a temporary file in the same directory and renaming it over the old one, to a new module in `cli/src/`. Verify with these unit tests: + - Tests that build equivalent nested values with keys inserted in different orders, serialize each, and assert identical text with keys sorted at every level, and a test that `commit.tags`, `repo.tags` and `requires.secrets` given in different orders serialize to the same text. - A test that updating one destination leaves the others unchanged. - A test that a record write that fails before the rename leaves the earlier record file unchanged, its pending entries included. - A test that reading a file whose `format_version` is 2, or whose `kind` names the other resource, returns an error naming the field and its value. - [ ] 4.2 Add the JSON comparison, which compares every key under the `repo`, `commit` and `assistant` objects except `commit.model_config` and `commit.model_provider`, and reports differences by JSON Pointer. Verify with these tests: - - Unit tests on nested objects, arrays and missing keys. + - Unit tests on nested objects, arrays, missing keys, and keys holding `/` or `~`, which the report escapes as RFC 6901 requires, so `a/b` appears as `/a~1b` and `a~b` as `/a~0b`. - A test that a changed `id` array inside `commit.manifest`, such as `ChatOpenAI` replaced by `ChatAnthropic`, is reported at its path. - A test that a destination whose `commit.model_config` and `commit.model_provider` differ from the file's, while the manifest matches, is reported as a match. - [ ] 4.3 Add the shared result report and the exit statuses 0, 3 and 1, printing JSON to stdout and progress to stderr, and leave clap's status 2 for usage errors; verify with unit tests of the report and of the exit status each result gives @@ -110,12 +110,12 @@ Each integration test in this group carries `#[cfg_attr(not(feature = "integrati ## 9. assistant compare and verify -- [ ] 9.1 Add `langstar assistant compare` with `--assistant-id`, the lookup of the recorded assistant in the destination record from 4.1, and the check, shared by assistant compare, verify and import, that each pointer under `requires.prompts` still resolves in the file's `assistant` object to the prompt name recorded with it. Verify with these tests: +- [ ] 9.1 Add `langstar assistant compare` with `--assistant-id`, `--record`, the lookup of the recorded assistant in the destination record from 4.1, and the check, shared by assistant compare, verify and import, that each pointer under `requires.prompts` still resolves in the file's `assistant` object to the prompt name recorded with it. Verify with these tests: - An integration test that exports an assistant, changes its metadata through the SDK, runs compare with `--assistant-id` set to that assistant, and asserts the reported path and exit status 3. - - A unit test that compare reads the assistant ID from a destination record written by the 4.1 types. + - Unit tests that compare reads the assistant ID from a destination record written by the 4.1 types, once at the default path and once at a path passed with `--record`. - An integration test that `assistant compare` with no destination record entry and no `--assistant-id` exits with status 1 and names the deployment. - A unit test that a file whose configuration names another prompt at a pointer under `requires.prompts` exits with status 1 and names the pointer and both values. -- [ ] 9.2 Add `langstar assistant verify` with `--assistant-id`, `--workspace-id` for the workspace that holds the prompts, the graph check, and the prompt check, which reads `GET /api/v1/commits/{owner}/{repo}/{tag}`. Verify with these tests: +- [ ] 9.2 Add `langstar assistant verify` with `--assistant-id`, `--record`, `--workspace-id` for the workspace that holds the prompts, the graph check, and the prompt check, which reads `GET /api/v1/commits/{owner}/{repo}/{tag}`. Verify with these tests: - An integration test that runs verify with `--assistant-id` against an exported assistant and exits with status 0. - An integration test that exports an assistant, changes its metadata through the SDK, runs verify with `--assistant-id` set to that assistant, and asserts the reported path and exit status 3. - An integration test of a file whose graph ID the deployment lacks, which reports the graph ID and exits with status 3. From 3aabe867997df6619a12bda81e1af23f7e5926e0 Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Wed, 14 Oct 2026 02:47:00 -0400 Subject: [PATCH 55/90] =?UTF-8?q?=F0=9F=93=9A=20docs(openspec):=20lead=20t?= =?UTF-8?q?he=20structured=20output=20check=20with=20its=20reason,=20name?= =?UTF-8?q?=20every=20sorted=20list=20in=20the=20export,=20and=20drop=20th?= =?UTF-8?q?e=20unsourced=20claim=20about=20how=20users=20run=20prompts?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit #807 Export, import, compare, and verify prompts and assistants across LangSmith environments Co-Authored-By: Claude Opus 5.5 --- .../changes/export-import-prompts-and-assistants/design.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/openspec/changes/export-import-prompts-and-assistants/design.md b/openspec/changes/export-import-prompts-and-assistants/design.md index 9bdd70554..a35ae5315 100644 --- a/openspec/changes/export-import-prompts-and-assistants/design.md +++ b/openspec/changes/export-import-prompts-and-assistants/design.md @@ -49,7 +49,7 @@ One prompt export looks like this: An assistant export has `kind: "assistant"`, a `source` with the deployment, assistant ID and version, an `assistant` object with `name`, `description`, `graph_id`, `config`, `context` and `metadata`, and `requires.prompts`. -The file holds no export time, and langstar sorts its keys and the commit's tags, so exporting an unchanged resource leaves Git with nothing to commit. +The file holds no export time, and langstar sorts its keys, the commit's tags, the repo's tags and the secret names under `requires.secrets`, so exporting an unchanged resource leaves Git with nothing to commit. Without `--output`, export prints the file to stdout and nothing else, so `langstar prompt export my-prompt > my-prompt.json` writes a file import can read. Export writes its progress messages to stderr in every output format. [`OutputFormatter::print_message` in cli/src/output.rs:173](https://github.com/codekiln/langstar/blob/e4ae746d51f1f3f9ccf3fc97787148c3b4df2bce/cli/src/output.rs#L173), which `info` calls, prints to stdout in table mode, so export does not use it for those messages. @@ -107,7 +107,7 @@ Status 0 means the resource matches and every check passed. Status 3 means langs ### A Python script checks that an imported structured prompt still returns structured data -A mise task runs a short Python script with `uv run --script`, which installs the `langsmith` and `langchain-openai` packages the script names as inline dependencies, locally and in the Integration Tests CI job. The script pulls the imported prompt with `include_model=True`, invokes it, and checks that the result matches the prompt's schema. The script passes `secrets={"OPENAI_API_KEY": }` to [`pull_prompt`](https://github.com/langchain-ai/langsmith-sdk/blob/47c553caf08c1989e2be3ff1886da3eb801f709f/python/langsmith/client.py#L10228), because the LangSmith Python SDK reads no secret from the environment unless asked and raises when a manifest holds a secret reference and no `secrets` map arrives. The script exists because the LangSmith API has no documented way to run a prompt with its model: the one prompt endpoint in the OpenAPI spec, `POST /api/v1/prompts/invoke_prompt`, takes message templates and inputs with no model or schema, and no LangSmith SDK calls it. In [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8), a developer calls that endpoint on the live API in case it does more than the spec says. The script needs an OpenAI API key. codekiln creates it and stores it in 1Password, in the repository's CI secrets and as the destination workspace's `OPENAI_API_KEY` secret, and in [export-import-prompts-and-assistants Task 2.1 Add the destination workspace and the model provider key](tasks.md#L16) a developer adds it to `fnox.toml` and the CI workflow. +The LangSmith API has no documented way to run a prompt with its model, so a short Python script runs each imported structured prompt with LangChain and checks that the result matches the prompt's schema. The one prompt endpoint in the OpenAPI spec, `POST /api/v1/prompts/invoke_prompt`, takes message templates and inputs with no model or schema, and no LangSmith SDK calls it; in [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8), a developer calls it on the live API in case it does more than the spec says. A mise task runs the script with `uv run --script`, which installs the `langsmith` and `langchain-openai` packages the script lists, on a developer's machine and in the Integration Tests CI job. The script pulls the imported prompt with `include_model=True` and passes `secrets={"OPENAI_API_KEY": }` to [`pull_prompt`](https://github.com/langchain-ai/langsmith-sdk/blob/47c553caf08c1989e2be3ff1886da3eb801f709f/python/langsmith/client.py#L10228), because the LangSmith Python SDK reads no secret from the environment unless asked, and raises when a manifest holds a secret reference and no `secrets` map arrives. The script needs an OpenAI API key. codekiln creates it and stores it in 1Password, in the repository's CI secrets and as the destination workspace's `OPENAI_API_KEY` secret, and in [export-import-prompts-and-assistants Task 2.1 Add the destination workspace and the model provider key](tasks.md#L16) a developer adds it to `fnox.toml` and the CI workflow. ### Export and import use the same SDK calls as the planned `prompt create`, `get` and `update` @@ -147,7 +147,7 @@ None. ### 3 - How should the end-to-end test show that an imported structured prompt still returns structured data? > Context from the drafting agent, for question 3. -> The LangSmith OpenAPI spec's only prompt-invoke endpoint, `POST /api/v1/prompts/invoke_prompt`, takes message templates and inputs with no model or schema, and no SDK calls it. In [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8), a developer calls it on the live API. The draft runs a Python script through a mise task with `uv run --script`, using the LangSmith Python SDK and LangChain, which is how most users run a prompt; the script names both as inline dependencies, and [export-import-prompts-and-assistants Task 7.3 Add the structured output check](tasks.md#L97) adds `uv` to `mise.toml` and removes `install: false` from the Integration Tests CI job, so that job runs `mise install`. It needs a model provider key, such as `OPENAI_API_KEY`, which codekiln would create and store in 1Password, the CI secrets and the destination workspace's secrets. Building the model call into langstar as `prompt verify --invoke` would need langstar to call each provider's API itself. Recommendation: the Python script, with an OpenAI key and a small model. +> The LangSmith OpenAPI spec's only prompt-invoke endpoint, `POST /api/v1/prompts/invoke_prompt`, takes message templates and inputs with no model or schema, and no SDK calls it. In [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8), a developer calls it on the live API. The draft runs a Python script through a mise task with `uv run --script`. The script pulls the prompt with the LangSmith Python SDK's `pull_prompt`, runs it with LangChain, and names both packages as inline dependencies. [export-import-prompts-and-assistants Task 7.3 Add the structured output check](tasks.md#L97) adds `uv` to `mise.toml` and removes `install: false` from the Integration Tests CI job, so that job runs `mise install`. It needs a model provider key, such as `OPENAI_API_KEY`, which codekiln would create and store in 1Password, the CI secrets and the destination workspace's secrets. Building the model call into langstar as `prompt verify --invoke` would need langstar to call each provider's API itself. Recommendation: the Python script, with an OpenAI key and a small model. From 4a240ea4b71a42030097fd2f8d9215d66ef17915 Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Wed, 14 Oct 2026 03:04:45 -0400 Subject: [PATCH 56/90] =?UTF-8?q?=F0=9F=93=9A=20docs(openspec):=20open=20t?= =?UTF-8?q?he=20assistant=20Context=20paragraph=20with=20the=20SDK=20gap?= =?UTF-8?q?=20the=20export=20needs?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit #807 Export, import, compare, and verify prompts and assistants across LangSmith environments Co-Authored-By: Claude Opus 5.5 --- openspec/changes/export-import-prompts-and-assistants/design.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/openspec/changes/export-import-prompts-and-assistants/design.md b/openspec/changes/export-import-prompts-and-assistants/design.md index a35ae5315..c806a7046 100644 --- a/openspec/changes/export-import-prompts-and-assistants/design.md +++ b/openspec/changes/export-import-prompts-and-assistants/design.md @@ -4,7 +4,7 @@ `langstar prompt push` in `cli/src/commands/prompt.rs` builds one shape of prompt: a single human message, with an optional schema from [#402 structured output prompts](https://github.com/codekiln/langstar/issues/402). `langstar prompt pull` prints any manifest LangSmith returns, but `PromptClient` in `sdk/src/prompts.rs` never asks LangSmith for the model, so that manifest has none. `PromptClient` keeps the manifest as raw JSON and has a typed `StructuredPrompt` view that holds messages, the schema and the method. -`langstar assistant` in `cli/src/commands/assistant.rs` takes `--deployment` on every subcommand and sends its requests to the Agent Server URL that `resolve_deployment_url` in `cli/src/deployment_utils.rs` reads from the deployment's `source_config.custom_url`. The control plane fills in `source_config.custom_url` for a deployment built from GitHub, as `SourceConfig.custom_url` in `reference/openapi/langchain/control-plane/openapi.json` says, so `langstar assistant` already reaches the test deployment: `test_assistant_description_field` in `cli/tests/assistant_command_test.rs` creates and reads an assistant there on every CI run. `AssistantClient` in `sdk/src/assistants.rs` has no `context` or `version` field and its search cannot filter by graph ID. +Langstar's assistant SDK lacks what an export needs: `Assistant` in `sdk/src/assistants.rs` has no `context` or `version` field, and the search in `AssistantClient` cannot filter by graph ID. The assistant commands themselves already work against the test deployment: `test_assistant_description_field` in `cli/tests/assistant_command_test.rs` creates and reads an assistant there in the Integration Tests CI job. Langstar reads one API key and one workspace at a time, from flags, environment variables or the config file. Langstar exits with status 1 when a command fails, and clap, the library that parses langstar's flags, exits with status 2 when it rejects a mistyped flag or a missing required one. From 360013c87314e38ddd8c2c267daff46770db5cd7 Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Wed, 14 Oct 2026 03:09:45 -0400 Subject: [PATCH 57/90] =?UTF-8?q?=F0=9F=93=9A=20docs(openspec):=20test=20c?= =?UTF-8?q?ompare=20--commit=20against=20an=20older=20tag=20and=20--update?= =?UTF-8?q?=20creating=20missing=20commit=20tags?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Copilot listed both as previously missed. Update the design's task 7.3 and 8.2 links for the two added lines. #807 Export, import, compare, and verify prompts and assistants across LangSmith environments Co-Authored-By: Claude Opus 5.5 --- .../changes/export-import-prompts-and-assistants/design.md | 4 ++-- .../changes/export-import-prompts-and-assistants/tasks.md | 2 ++ 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/openspec/changes/export-import-prompts-and-assistants/design.md b/openspec/changes/export-import-prompts-and-assistants/design.md index c806a7046..3c301be02 100644 --- a/openspec/changes/export-import-prompts-and-assistants/design.md +++ b/openspec/changes/export-import-prompts-and-assistants/design.md @@ -119,7 +119,7 @@ The LangSmith API has no documented way to run a prompt with its model, so a sho - [Pushing a manifest identical to the latest commit may return 409] → Import compares first and pushes nothing when the prompt matches. - [The SDK changes for export and import add public fields to `Prompt`, `CommitManifestResponse`, `CommitData`, `CommitRequest`, `Assistant` and the assistant request structs, so a crate that builds one of those structs with a struct literal or destructures it with a pattern that names every field stops compiling] → Each pull request that adds a field updates every struct literal of that struct in this repository. [Should the SDK field additions count as a breaking change?](#9---should-the-sdk-field-additions-count-as-a-breaking-change) asks codekiln whether those pull requests also carry a `BREAKING CHANGE:` footer, which `docs/dev/git-scm-conventions.md` says makes the release a major version. - [A prompt's manifest may hold a credential written as a literal model setting, rather than as a secret reference] → Export refuses with status 1 and writes nothing when a chat model setting whose name ends in `api_key` holds a string, and names that setting's path without its value. A credential under any other name, such as a header, still reaches the file, because langstar cannot tell it from any other string. The prompt export docs in [export-import-prompts-and-assistants Task 5.2 Document prompt export](tasks.md#L56) tell users to keep credentials in the workspace's secrets and to read an export file before committing it. -- [An assistant's configuration, context or metadata may hold a credential, which export copies into a file meant for Git] → Export writes those objects as the Agent Server returns them, since a credential there looks like any other string. The assistant export docs in [export-import-prompts-and-assistants Task 8.2 Document assistant export](tasks.md#L109) tell users to keep credentials in the deployment's environment variables or the workspace's secrets, and to read an export file before committing it. +- [An assistant's configuration, context or metadata may hold a credential, which export copies into a file meant for Git] → Export writes those objects as the Agent Server returns them, since a credential there looks like any other string. The assistant export docs in [export-import-prompts-and-assistants Task 8.2 Document assistant export](tasks.md#L111) tell users to keep credentials in the deployment's environment variables or the workspace's secrets, and to read an export file before committing it. - [A tag in the destination may already point at another commit] → `--update` moves the tag, and `--dry-run` lists the move. - [`--update` deletes a tag on the matching destination commit that the file lacks, and a graph on a deployment may read that tag] → `--dry-run` lists each delete, and `assistant verify` reports a prompt tag an assistant refers to that the workspace no longer has. - [The control plane may refuse a second deployment from this repository with 409] → In [export-import-prompts-and-assistants Task 2.3 Register the test graph under a second graph ID](tasks.md#L18), a developer deploys the new `langgraph.json` once, before anyone builds the fixture. If the control plane answers with 409, the developer brings [Which second workspace and second deployment should the tests use?](#5---which-second-workspace-and-second-deployment-should-the-tests-use) back to codekiln. @@ -147,7 +147,7 @@ None. ### 3 - How should the end-to-end test show that an imported structured prompt still returns structured data? > Context from the drafting agent, for question 3. -> The LangSmith OpenAPI spec's only prompt-invoke endpoint, `POST /api/v1/prompts/invoke_prompt`, takes message templates and inputs with no model or schema, and no SDK calls it. In [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8), a developer calls it on the live API. The draft runs a Python script through a mise task with `uv run --script`. The script pulls the prompt with the LangSmith Python SDK's `pull_prompt`, runs it with LangChain, and names both packages as inline dependencies. [export-import-prompts-and-assistants Task 7.3 Add the structured output check](tasks.md#L97) adds `uv` to `mise.toml` and removes `install: false` from the Integration Tests CI job, so that job runs `mise install`. It needs a model provider key, such as `OPENAI_API_KEY`, which codekiln would create and store in 1Password, the CI secrets and the destination workspace's secrets. Building the model call into langstar as `prompt verify --invoke` would need langstar to call each provider's API itself. Recommendation: the Python script, with an OpenAI key and a small model. +> The LangSmith OpenAPI spec's only prompt-invoke endpoint, `POST /api/v1/prompts/invoke_prompt`, takes message templates and inputs with no model or schema, and no SDK calls it. In [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8), a developer calls it on the live API. The draft runs a Python script through a mise task with `uv run --script`. The script pulls the prompt with the LangSmith Python SDK's `pull_prompt`, runs it with LangChain, and names both packages as inline dependencies. [export-import-prompts-and-assistants Task 7.3 Add the structured output check](tasks.md#L99) adds `uv` to `mise.toml` and removes `install: false` from the Integration Tests CI job, so that job runs `mise install`. It needs a model provider key, such as `OPENAI_API_KEY`, which codekiln would create and store in 1Password, the CI secrets and the destination workspace's secrets. Building the model call into langstar as `prompt verify --invoke` would need langstar to call each provider's API itself. Recommendation: the Python script, with an OpenAI key and a small model. diff --git a/openspec/changes/export-import-prompts-and-assistants/tasks.md b/openspec/changes/export-import-prompts-and-assistants/tasks.md index 402d49219..d9e696fab 100644 --- a/openspec/changes/export-import-prompts-and-assistants/tasks.md +++ b/openspec/changes/export-import-prompts-and-assistants/tasks.md @@ -61,6 +61,7 @@ Each integration test in this group carries `#[cfg_attr(not(feature = "integrati - An integration test that exports a prompt, changes its model temperature through the SDK, and asserts the reported path, both values and exit status 3. - Integration tests that a changed commit description and a changed repo readme are each reported the same way. - An integration test that exports a prompt, adds a tag to the exported commit through the SDK, and asserts that compare reports `/commit/tags` with both values and exits with status 3. + - An integration test that exports a tagged commit, pushes a newer commit to the prompt through the SDK, runs `prompt compare --commit `, and asserts that it compares the tagged commit and exits with status 0. - A CLI test that `langstar -f json prompt compare` prints one JSON object with `status`, `differences` and `checks` to stdout and its progress to stderr. - A CLI test that `prompt compare` with a mistyped flag exits with status 2. - A CLI test that `prompt compare` of a file holding `"format_version": 2` exits with status 1 and names `format_version` and the value 2. @@ -89,6 +90,7 @@ Each integration test in this group carries `#[cfg_attr(not(feature = "integrati - An integration test that pushes the file's manifest to the destination, leaves an older commit hash in the destination record, imports the file, and asserts that import exits with status 0, prints no change warning and leaves the latest commit's hash in the record. - An integration test that pushes the file's manifest and tags to the destination, imports the file with no destination record, and asserts that import exits with status 0 and writes the latest commit's hash to the record under the destination workspace. - An integration test that `--update` adds one commit with the right parent and the file's commit description. + - An integration test that pushes the file's manifest to the destination without the file's tags, as an import that stopped before creating its tags leaves it, runs import with `--update`, and asserts that it creates the missing tags on that commit and adds no commit. - An integration test that `--update` against a destination whose repo description, readme, tags or visibility differ while the manifest matches updates the repo and adds no commit. - An integration test that `--update` against a destination whose `prod` tag points at an earlier commit while the manifest matches moves the tag and adds no commit, and that `--dry-run` lists the move. - An integration test that `--update` against a destination whose latest commit matches the file and carries a `staging` tag the file lacks deletes `staging`, adds no commit and exits with status 0, and that `--dry-run` lists the delete. From 18a8855b2d6636719ca4d62dce360e243890e445 Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Wed, 14 Oct 2026 03:14:46 -0400 Subject: [PATCH 58/90] =?UTF-8?q?=F0=9F=93=9A=20docs(openspec):=20redact?= =?UTF-8?q?=20a=20literal=20API=20key=20in=20compare=20output,=20and=20poi?= =?UTF-8?q?nt=20the=20structured=20output=20check=20at=20the=20destination?= =?UTF-8?q?=20workspace?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit #807 Export, import, compare, and verify prompts and assistants across LangSmith environments Co-Authored-By: Claude Opus 5.5 --- .../export-import-prompts-and-assistants/design.md | 6 +++--- .../specs/prompt/spec.md | 10 ++++++++++ .../export-import-prompts-and-assistants/tasks.md | 3 ++- 3 files changed, 15 insertions(+), 4 deletions(-) diff --git a/openspec/changes/export-import-prompts-and-assistants/design.md b/openspec/changes/export-import-prompts-and-assistants/design.md index 3c301be02..2e3b72864 100644 --- a/openspec/changes/export-import-prompts-and-assistants/design.md +++ b/openspec/changes/export-import-prompts-and-assistants/design.md @@ -118,8 +118,8 @@ The LangSmith API has no documented way to run a prompt with its model, so a sho - [LangSmith may store a pushed model differently from the source] → Import reads the prompt back and reports any difference, and in [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8) a developer checks the round trip against the live API before anyone builds the commands. - [Pushing a manifest identical to the latest commit may return 409] → Import compares first and pushes nothing when the prompt matches. - [The SDK changes for export and import add public fields to `Prompt`, `CommitManifestResponse`, `CommitData`, `CommitRequest`, `Assistant` and the assistant request structs, so a crate that builds one of those structs with a struct literal or destructures it with a pattern that names every field stops compiling] → Each pull request that adds a field updates every struct literal of that struct in this repository. [Should the SDK field additions count as a breaking change?](#9---should-the-sdk-field-additions-count-as-a-breaking-change) asks codekiln whether those pull requests also carry a `BREAKING CHANGE:` footer, which `docs/dev/git-scm-conventions.md` says makes the release a major version. -- [A prompt's manifest may hold a credential written as a literal model setting, rather than as a secret reference] → Export refuses with status 1 and writes nothing when a chat model setting whose name ends in `api_key` holds a string, and names that setting's path without its value. A credential under any other name, such as a header, still reaches the file, because langstar cannot tell it from any other string. The prompt export docs in [export-import-prompts-and-assistants Task 5.2 Document prompt export](tasks.md#L56) tell users to keep credentials in the workspace's secrets and to read an export file before committing it. -- [An assistant's configuration, context or metadata may hold a credential, which export copies into a file meant for Git] → Export writes those objects as the Agent Server returns them, since a credential there looks like any other string. The assistant export docs in [export-import-prompts-and-assistants Task 8.2 Document assistant export](tasks.md#L111) tell users to keep credentials in the deployment's environment variables or the workspace's secrets, and to read an export file before committing it. +- [A prompt's manifest may hold a credential written as a literal model setting, rather than as a secret reference] → Export refuses with status 1 and writes nothing when a chat model setting whose name ends in `api_key` holds a string, and names that setting's path without its value. A credential under any other name, such as a header, still reaches the file, because langstar cannot tell it from any other string. The prompt export docs in [export-import-prompts-and-assistants Task 5.2 Document prompt export](tasks.md#L57) tell users to keep credentials in the workspace's secrets and to read an export file before committing it. +- [An assistant's configuration, context or metadata may hold a credential, which export copies into a file meant for Git] → Export writes those objects as the Agent Server returns them, since a credential there looks like any other string. The assistant export docs in [export-import-prompts-and-assistants Task 8.2 Document assistant export](tasks.md#L112) tell users to keep credentials in the deployment's environment variables or the workspace's secrets, and to read an export file before committing it. - [A tag in the destination may already point at another commit] → `--update` moves the tag, and `--dry-run` lists the move. - [`--update` deletes a tag on the matching destination commit that the file lacks, and a graph on a deployment may read that tag] → `--dry-run` lists each delete, and `assistant verify` reports a prompt tag an assistant refers to that the workspace no longer has. - [The control plane may refuse a second deployment from this repository with 409] → In [export-import-prompts-and-assistants Task 2.3 Register the test graph under a second graph ID](tasks.md#L18), a developer deploys the new `langgraph.json` once, before anyone builds the fixture. If the control plane answers with 409, the developer brings [Which second workspace and second deployment should the tests use?](#5---which-second-workspace-and-second-deployment-should-the-tests-use) back to codekiln. @@ -147,7 +147,7 @@ None. ### 3 - How should the end-to-end test show that an imported structured prompt still returns structured data? > Context from the drafting agent, for question 3. -> The LangSmith OpenAPI spec's only prompt-invoke endpoint, `POST /api/v1/prompts/invoke_prompt`, takes message templates and inputs with no model or schema, and no SDK calls it. In [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8), a developer calls it on the live API. The draft runs a Python script through a mise task with `uv run --script`. The script pulls the prompt with the LangSmith Python SDK's `pull_prompt`, runs it with LangChain, and names both packages as inline dependencies. [export-import-prompts-and-assistants Task 7.3 Add the structured output check](tasks.md#L99) adds `uv` to `mise.toml` and removes `install: false` from the Integration Tests CI job, so that job runs `mise install`. It needs a model provider key, such as `OPENAI_API_KEY`, which codekiln would create and store in 1Password, the CI secrets and the destination workspace's secrets. Building the model call into langstar as `prompt verify --invoke` would need langstar to call each provider's API itself. Recommendation: the Python script, with an OpenAI key and a small model. +> The LangSmith OpenAPI spec's only prompt-invoke endpoint, `POST /api/v1/prompts/invoke_prompt`, takes message templates and inputs with no model or schema, and no SDK calls it. In [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8), a developer calls it on the live API. The draft runs a Python script through a mise task with `uv run --script`. The script pulls the prompt with the LangSmith Python SDK's `pull_prompt`, runs it with LangChain, and names both packages as inline dependencies. [export-import-prompts-and-assistants Task 7.3 Add the structured output check](tasks.md#L100) adds `uv` to `mise.toml` and removes `install: false` from the Integration Tests CI job, so that job runs `mise install`. It needs a model provider key, such as `OPENAI_API_KEY`, which codekiln would create and store in 1Password, the CI secrets and the destination workspace's secrets. Building the model call into langstar as `prompt verify --invoke` would need langstar to call each provider's API itself. Recommendation: the Python script, with an OpenAI key and a small model. diff --git a/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md index 3acbbd4b8..1876f6ea9 100644 --- a/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md +++ b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md @@ -249,6 +249,16 @@ After writing, `langstar prompt import` SHALL record the commit hash in the dest - **THEN** langstar lists that field as a difference and exits with status 3 - **AND** the destination record holds the new commit hash +### Requirement: Redact an API key written into a destination prompt's model + +When `prompt compare`, `prompt verify` or `prompt import` reports a difference at a chat model setting whose name ends in `api_key` and either side holds a string there, langstar SHALL print `` in place of that string in the text and JSON reports. + +#### Scenario: Destination model holds a literal API key + +- **WHEN** someone saved the destination prompt with an `openai_api_key` written as a string +- **AND** a user runs `langstar prompt compare my-prompt.json` +- **THEN** langstar reports the setting's path with `` in place of the string and exits with status 3 + ### Requirement: Compare a prompt with its export file `langstar prompt compare ` SHALL read the prompt with the file's handle from the workspace, with its model and repo details, and list each difference by its JSON path with both values. `--commit ` SHALL choose the commit, and without it the latest. It SHALL compare every field in the file's `repo` and `commit`, including every manifest key and its `id` arrays, except `commit.model_config` and `commit.model_provider`, which import cannot set. diff --git a/openspec/changes/export-import-prompts-and-assistants/tasks.md b/openspec/changes/export-import-prompts-and-assistants/tasks.md index d9e696fab..c274033b4 100644 --- a/openspec/changes/export-import-prompts-and-assistants/tasks.md +++ b/openspec/changes/export-import-prompts-and-assistants/tasks.md @@ -40,6 +40,7 @@ Each integration test in this group carries `#[cfg_attr(not(feature = "integrati - Unit tests on nested objects, arrays, missing keys, and keys holding `/` or `~`, which the report escapes as RFC 6901 requires, so `a/b` appears as `/a~1b` and `a~b` as `/a~0b`. - A test that a changed `id` array inside `commit.manifest`, such as `ChatOpenAI` replaced by `ChatAnthropic`, is reported at its path. - A test that a destination whose `commit.model_config` and `commit.model_provider` differ from the file's, while the manifest matches, is reported as a match. + - A test that a destination manifest whose `openai_api_key` holds a string reports that path with `` in place of the string, in both the text and the `-f json` report. - [ ] 4.3 Add the shared result report and the exit statuses 0, 3 and 1, printing JSON to stdout and progress to stderr, and leave clap's status 2 for usage errors; verify with unit tests of the report and of the exit status each result gives - [ ] 4.4 Add the secret reference walk that lists secret names from a manifest and returns the JSON Pointer path of each chat model setting whose name ends in `api_key` and holds a string instead of a secret reference, which `prompt export` from 5.1 refuses with status 1; verify with unit tests on a copy of the `prompt_with_model` manifest from the LangSmith Python SDK's integration tests, which `references/langsmith-sdk.md` links, on a manifest with no secret reference, and on a manifest whose `openai_api_key` holds a string, asserting that the walk returns that setting's path and that the error message holds the path and not the string @@ -96,7 +97,7 @@ Each integration test in this group carries `#[cfg_attr(not(feature = "integrati - An integration test that `--update` against a destination whose latest commit matches the file and carries a `staging` tag the file lacks deletes `staging`, adds no commit and exits with status 0, and that `--dry-run` lists the delete. - An integration test that `--dry-run` writes nothing. - A unit test in `cli/src/commands/prompt.rs` that serves, through a `mockito` server and `LangchainClient::with_base_urls`, a commit read back that lacks a field the file holds, calls the import code, and asserts that the result names that field's JSON Pointer path and gives exit status 3, and that the destination record holds the new commit hash. -- [ ] 7.3 Add the structured output check as a mise task that runs a Python script with `uv run --script`. The script names `langsmith` and `langchain-openai` as inline script dependencies, pulls the imported prompt with `include_model=True` and a `secrets` map holding the model provider key read from the environment, invokes it and validates the result against the schema. Add `uv` to `mise.toml`, and remove `install: false` from the Setup mise step of the Integration Tests job in `.github/workflows/ci.yml` so that job installs `uv` and Python 3.11. Verify by having the import integration test create a `json_schema` prompt and a `function_calling` prompt, each saved with a small OpenAI model, import both into the destination workspace and run the check on each, under `mise run check` and in that CI job. +- [ ] 7.3 Add the structured output check as a mise task that runs a Python script with `uv run --script`. The script names `langsmith` and `langchain-openai` as inline script dependencies, builds its `Client` with `workspace_id` set to `LANGSTAR_TEST_DEST_WORKSPACE_ID`, so it reads the destination copy even though CI sets `LANGSMITH_WORKSPACE_ID` to the source workspace, pulls the imported prompt with `include_model=True` and a `secrets` map holding the model provider key read from the environment, invokes it and validates the result against the schema. Add `uv` to `mise.toml`, and remove `install: false` from the Setup mise step of the Integration Tests job in `.github/workflows/ci.yml` so that job installs `uv` and Python 3.11. Verify by having the import integration test create a `json_schema` prompt and a `function_calling` prompt, each saved with a small OpenAI model, import both into the destination workspace and run the check on each, under `mise run check` and in that CI job. - [ ] 7.4 Document prompt import, the workspace scope of prompts and setting up destination credentials; verify that each documented command runs as written ## 8. assistant export From 2bcd529526f3cd439a5809a6293a5e0e943d2171 Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Wed, 14 Oct 2026 03:23:22 -0400 Subject: [PATCH 59/90] =?UTF-8?q?=F0=9F=93=9A=20docs(openspec):=20redact?= =?UTF-8?q?=20API=20keys=20in=20the=20shared=20report=20task,=20say=20comp?= =?UTF-8?q?are=20redacts=20them,=20and=20name=20the=20destination=20worksp?= =?UTF-8?q?ace=20in=20the=20structured=20output=20check?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit #807 Export, import, compare, and verify prompts and assistants across LangSmith environments Co-Authored-By: Claude Opus 5.5 --- .../export-import-prompts-and-assistants/design.md | 10 +++++----- .../export-import-prompts-and-assistants/tasks.md | 5 ++--- 2 files changed, 7 insertions(+), 8 deletions(-) diff --git a/openspec/changes/export-import-prompts-and-assistants/design.md b/openspec/changes/export-import-prompts-and-assistants/design.md index 2e3b72864..629fb49d0 100644 --- a/openspec/changes/export-import-prompts-and-assistants/design.md +++ b/openspec/changes/export-import-prompts-and-assistants/design.md @@ -81,7 +81,7 @@ Without `--update`, import lists the differences and exits with status 3. `--dry ### Compare reports differences by JSON path -Compare reads the prompt or assistant from the destination, puts it in the shape of the export file's `repo`, `commit` and `assistant` objects, and lists every value that differs from the file. Compare checks every key in those objects, including the chat model's settings and the `id` arrays inside a manifest that name the prompt and model classes, such as `ChatOpenAI`. It skips `commit.model_config` and `commit.model_provider`, because LangSmith fills both in itself when import pushes a commit, and a destination workspace can fill them in differently from the source. Export leaves the values the server sets, such as IDs, version numbers, timestamps, owner and like counts, out of those objects, so they never show up as differences. The source environment's workspace, deployment and IDs sit in `source`, so they never show up as differences against a copy. Each difference names its JSON Pointer path and both values. For a prompt, the comparison covers the commit description, the commit tags and the commit each one points at, any tag on the compared commit that the file lacks, and the repo's description, readme, tags and visibility as well as the manifest. Import runs the same comparison after it writes, so a field LangSmith or the Agent Server drops shows up as a difference. +Compare reads the prompt or assistant from the destination, puts it in the shape of the export file's `repo`, `commit` and `assistant` objects, and lists every value that differs from the file. Compare checks every key in those objects, including the chat model's settings and the `id` arrays inside a manifest that name the prompt and model classes, such as `ChatOpenAI`. It skips `commit.model_config` and `commit.model_provider`, because LangSmith fills both in itself when import pushes a commit, and a destination workspace can fill them in differently from the source. Export leaves the values the server sets, such as IDs, version numbers, timestamps, owner and like counts, out of those objects, so they never show up as differences. The source environment's workspace, deployment and IDs sit in `source`, so they never show up as differences against a copy. Each difference names its JSON Pointer path and both values, except at a chat model setting whose name ends in `api_key`: there the report prints `` in place of a string on either side, so a key someone typed into the destination prompt's model never reaches a terminal or a CI log. For a prompt, the comparison covers the commit description, the commit tags and the commit each one points at, any tag on the compared commit that the file lacks, and the repo's description, readme, tags and visibility as well as the manifest. Import runs the same comparison after it writes, so a field LangSmith or the Agent Server drops shows up as a difference. ### Verify checks what the copy needs to run @@ -107,7 +107,7 @@ Status 0 means the resource matches and every check passed. Status 3 means langs ### A Python script checks that an imported structured prompt still returns structured data -The LangSmith API has no documented way to run a prompt with its model, so a short Python script runs each imported structured prompt with LangChain and checks that the result matches the prompt's schema. The one prompt endpoint in the OpenAPI spec, `POST /api/v1/prompts/invoke_prompt`, takes message templates and inputs with no model or schema, and no LangSmith SDK calls it; in [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8), a developer calls it on the live API in case it does more than the spec says. A mise task runs the script with `uv run --script`, which installs the `langsmith` and `langchain-openai` packages the script lists, on a developer's machine and in the Integration Tests CI job. The script pulls the imported prompt with `include_model=True` and passes `secrets={"OPENAI_API_KEY": }` to [`pull_prompt`](https://github.com/langchain-ai/langsmith-sdk/blob/47c553caf08c1989e2be3ff1886da3eb801f709f/python/langsmith/client.py#L10228), because the LangSmith Python SDK reads no secret from the environment unless asked, and raises when a manifest holds a secret reference and no `secrets` map arrives. The script needs an OpenAI API key. codekiln creates it and stores it in 1Password, in the repository's CI secrets and as the destination workspace's `OPENAI_API_KEY` secret, and in [export-import-prompts-and-assistants Task 2.1 Add the destination workspace and the model provider key](tasks.md#L16) a developer adds it to `fnox.toml` and the CI workflow. +The LangSmith API has no documented way to run a prompt with its model, so a short Python script runs each imported structured prompt with LangChain and checks that the result matches the prompt's schema. The one prompt endpoint in the OpenAPI spec, `POST /api/v1/prompts/invoke_prompt`, takes message templates and inputs with no model or schema, and no LangSmith SDK calls it; in [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8), a developer calls it on the live API in case it does more than the spec says. A mise task runs the script with `uv run --script`, which installs the `langsmith` and `langchain-openai` packages the script lists, on a developer's machine and in the Integration Tests CI job. The script reads the destination workspace: it builds its LangSmith `Client` with `workspace_id` set to `LANGSTAR_TEST_DEST_WORKSPACE_ID`, because CI sets `LANGSMITH_WORKSPACE_ID` to the source workspace, which holds a prompt with the same handle. The script pulls the imported prompt with `include_model=True` and passes `secrets={"OPENAI_API_KEY": }` to [`pull_prompt`](https://github.com/langchain-ai/langsmith-sdk/blob/47c553caf08c1989e2be3ff1886da3eb801f709f/python/langsmith/client.py#L10228), because the LangSmith Python SDK reads no secret from the environment unless asked, and raises when a manifest holds a secret reference and no `secrets` map arrives. The script needs an OpenAI API key. codekiln creates it and stores it in 1Password, in the repository's CI secrets and as the destination workspace's `OPENAI_API_KEY` secret, and in [export-import-prompts-and-assistants Task 2.1 Add the destination workspace and the model provider key](tasks.md#L16) a developer adds it to `fnox.toml` and the CI workflow. ### Export and import use the same SDK calls as the planned `prompt create`, `get` and `update` @@ -118,8 +118,8 @@ The LangSmith API has no documented way to run a prompt with its model, so a sho - [LangSmith may store a pushed model differently from the source] → Import reads the prompt back and reports any difference, and in [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8) a developer checks the round trip against the live API before anyone builds the commands. - [Pushing a manifest identical to the latest commit may return 409] → Import compares first and pushes nothing when the prompt matches. - [The SDK changes for export and import add public fields to `Prompt`, `CommitManifestResponse`, `CommitData`, `CommitRequest`, `Assistant` and the assistant request structs, so a crate that builds one of those structs with a struct literal or destructures it with a pattern that names every field stops compiling] → Each pull request that adds a field updates every struct literal of that struct in this repository. [Should the SDK field additions count as a breaking change?](#9---should-the-sdk-field-additions-count-as-a-breaking-change) asks codekiln whether those pull requests also carry a `BREAKING CHANGE:` footer, which `docs/dev/git-scm-conventions.md` says makes the release a major version. -- [A prompt's manifest may hold a credential written as a literal model setting, rather than as a secret reference] → Export refuses with status 1 and writes nothing when a chat model setting whose name ends in `api_key` holds a string, and names that setting's path without its value. A credential under any other name, such as a header, still reaches the file, because langstar cannot tell it from any other string. The prompt export docs in [export-import-prompts-and-assistants Task 5.2 Document prompt export](tasks.md#L57) tell users to keep credentials in the workspace's secrets and to read an export file before committing it. -- [An assistant's configuration, context or metadata may hold a credential, which export copies into a file meant for Git] → Export writes those objects as the Agent Server returns them, since a credential there looks like any other string. The assistant export docs in [export-import-prompts-and-assistants Task 8.2 Document assistant export](tasks.md#L112) tell users to keep credentials in the deployment's environment variables or the workspace's secrets, and to read an export file before committing it. +- [A prompt's manifest may hold a credential written as a literal model setting, rather than as a secret reference] → Export refuses with status 1 and writes nothing when a chat model setting whose name ends in `api_key` holds a string, and names that setting's path without its value. A credential under any other name, such as a header, still reaches the file, because langstar cannot tell it from any other string. The prompt export docs in [export-import-prompts-and-assistants Task 5.2 Document prompt export](tasks.md#L56) tell users to keep credentials in the workspace's secrets and to read an export file before committing it. +- [An assistant's configuration, context or metadata may hold a credential, which export copies into a file meant for Git] → Export writes those objects as the Agent Server returns them, since a credential there looks like any other string. The assistant export docs in [export-import-prompts-and-assistants Task 8.2 Document assistant export](tasks.md#L111) tell users to keep credentials in the deployment's environment variables or the workspace's secrets, and to read an export file before committing it. - [A tag in the destination may already point at another commit] → `--update` moves the tag, and `--dry-run` lists the move. - [`--update` deletes a tag on the matching destination commit that the file lacks, and a graph on a deployment may read that tag] → `--dry-run` lists each delete, and `assistant verify` reports a prompt tag an assistant refers to that the workspace no longer has. - [The control plane may refuse a second deployment from this repository with 409] → In [export-import-prompts-and-assistants Task 2.3 Register the test graph under a second graph ID](tasks.md#L18), a developer deploys the new `langgraph.json` once, before anyone builds the fixture. If the control plane answers with 409, the developer brings [Which second workspace and second deployment should the tests use?](#5---which-second-workspace-and-second-deployment-should-the-tests-use) back to codekiln. @@ -147,7 +147,7 @@ None. ### 3 - How should the end-to-end test show that an imported structured prompt still returns structured data? > Context from the drafting agent, for question 3. -> The LangSmith OpenAPI spec's only prompt-invoke endpoint, `POST /api/v1/prompts/invoke_prompt`, takes message templates and inputs with no model or schema, and no SDK calls it. In [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8), a developer calls it on the live API. The draft runs a Python script through a mise task with `uv run --script`. The script pulls the prompt with the LangSmith Python SDK's `pull_prompt`, runs it with LangChain, and names both packages as inline dependencies. [export-import-prompts-and-assistants Task 7.3 Add the structured output check](tasks.md#L100) adds `uv` to `mise.toml` and removes `install: false` from the Integration Tests CI job, so that job runs `mise install`. It needs a model provider key, such as `OPENAI_API_KEY`, which codekiln would create and store in 1Password, the CI secrets and the destination workspace's secrets. Building the model call into langstar as `prompt verify --invoke` would need langstar to call each provider's API itself. Recommendation: the Python script, with an OpenAI key and a small model. +> The LangSmith OpenAPI spec's only prompt-invoke endpoint, `POST /api/v1/prompts/invoke_prompt`, takes message templates and inputs with no model or schema, and no SDK calls it. In [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8), a developer calls it on the live API. The draft runs a Python script through a mise task with `uv run --script`. The script pulls the prompt from the destination workspace with the LangSmith Python SDK's `pull_prompt`, runs it with LangChain, and names both packages as inline dependencies. [export-import-prompts-and-assistants Task 7.3 Add the structured output check](tasks.md#L99) adds `uv` to `mise.toml` and removes `install: false` from the Integration Tests CI job, so that job runs `mise install`. It needs a model provider key, such as `OPENAI_API_KEY`, which codekiln would create and store in 1Password, the CI secrets and the destination workspace's secrets. Building the model call into langstar as `prompt verify --invoke` would need langstar to call each provider's API itself. Recommendation: the Python script, with an OpenAI key and a small model. diff --git a/openspec/changes/export-import-prompts-and-assistants/tasks.md b/openspec/changes/export-import-prompts-and-assistants/tasks.md index c274033b4..888031c62 100644 --- a/openspec/changes/export-import-prompts-and-assistants/tasks.md +++ b/openspec/changes/export-import-prompts-and-assistants/tasks.md @@ -40,8 +40,7 @@ Each integration test in this group carries `#[cfg_attr(not(feature = "integrati - Unit tests on nested objects, arrays, missing keys, and keys holding `/` or `~`, which the report escapes as RFC 6901 requires, so `a/b` appears as `/a~1b` and `a~b` as `/a~0b`. - A test that a changed `id` array inside `commit.manifest`, such as `ChatOpenAI` replaced by `ChatAnthropic`, is reported at its path. - A test that a destination whose `commit.model_config` and `commit.model_provider` differ from the file's, while the manifest matches, is reported as a match. - - A test that a destination manifest whose `openai_api_key` holds a string reports that path with `` in place of the string, in both the text and the `-f json` report. -- [ ] 4.3 Add the shared result report and the exit statuses 0, 3 and 1, printing JSON to stdout and progress to stderr, and leave clap's status 2 for usage errors; verify with unit tests of the report and of the exit status each result gives +- [ ] 4.3 Add the shared result report, which `prompt compare`, `prompt verify` and `prompt import` all print through, and the exit statuses 0, 3 and 1. The report prints JSON to stdout and progress to stderr, and prints `` in place of a string at any chat model setting whose name ends in `api_key`, on either side of a difference. Leave clap's status 2 for usage errors. Verify with unit tests of the report, of the exit status each result gives, and of a difference at `openai_api_key` where the destination holds a string, asserting that the text and the `-f json` report each print `` and neither holds the string. - [ ] 4.4 Add the secret reference walk that lists secret names from a manifest and returns the JSON Pointer path of each chat model setting whose name ends in `api_key` and holds a string instead of a secret reference, which `prompt export` from 5.1 refuses with status 1; verify with unit tests on a copy of the `prompt_with_model` manifest from the LangSmith Python SDK's integration tests, which `references/langsmith-sdk.md` links, on a manifest with no secret reference, and on a manifest whose `openai_api_key` holds a string, asserting that the walk returns that setting's path and that the error message holds the path and not the string ## 5. prompt export @@ -97,7 +96,7 @@ Each integration test in this group carries `#[cfg_attr(not(feature = "integrati - An integration test that `--update` against a destination whose latest commit matches the file and carries a `staging` tag the file lacks deletes `staging`, adds no commit and exits with status 0, and that `--dry-run` lists the delete. - An integration test that `--dry-run` writes nothing. - A unit test in `cli/src/commands/prompt.rs` that serves, through a `mockito` server and `LangchainClient::with_base_urls`, a commit read back that lacks a field the file holds, calls the import code, and asserts that the result names that field's JSON Pointer path and gives exit status 3, and that the destination record holds the new commit hash. -- [ ] 7.3 Add the structured output check as a mise task that runs a Python script with `uv run --script`. The script names `langsmith` and `langchain-openai` as inline script dependencies, builds its `Client` with `workspace_id` set to `LANGSTAR_TEST_DEST_WORKSPACE_ID`, so it reads the destination copy even though CI sets `LANGSMITH_WORKSPACE_ID` to the source workspace, pulls the imported prompt with `include_model=True` and a `secrets` map holding the model provider key read from the environment, invokes it and validates the result against the schema. Add `uv` to `mise.toml`, and remove `install: false` from the Setup mise step of the Integration Tests job in `.github/workflows/ci.yml` so that job installs `uv` and Python 3.11. Verify by having the import integration test create a `json_schema` prompt and a `function_calling` prompt, each saved with a small OpenAI model, import both into the destination workspace and run the check on each, under `mise run check` and in that CI job. +- [ ] 7.3 Add the structured output check as a mise task that runs a Python script with `uv run --script`. The script names `langsmith` and `langchain-openai` as inline script dependencies. It builds its `Client` with `workspace_id` set to `LANGSTAR_TEST_DEST_WORKSPACE_ID`, because CI sets `LANGSMITH_WORKSPACE_ID` to the source workspace, which holds a prompt with the same handle. It then pulls the imported prompt with `include_model=True` and a `secrets` map holding the model provider key read from the environment, invokes it and validates the result against the schema. Add `uv` to `mise.toml`, and remove `install: false` from the Setup mise step of the Integration Tests job in `.github/workflows/ci.yml` so that job installs `uv` and Python 3.11. Verify by having the import integration test create a `json_schema` prompt and a `function_calling` prompt, each saved with a small OpenAI model, import both into the destination workspace and run the check on each, under `mise run check` and in that CI job. - [ ] 7.4 Document prompt import, the workspace scope of prompts and setting up destination credentials; verify that each documented command runs as written ## 8. assistant export From 5023d6997122677afce34510fa944607bbbb773d Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Wed, 14 Oct 2026 03:34:35 -0400 Subject: [PATCH 60/90] =?UTF-8?q?=F0=9F=93=9A=20docs(openspec):=20redact?= =?UTF-8?q?=20API=20keys=20inside=20any=20printed=20difference=20value,=20?= =?UTF-8?q?name=20both=20resource=20types=20in=20the=20shared=20report=20t?= =?UTF-8?q?ask,=20and=20say=20fnox.toml=20also=20sets=20the=20source=20wor?= =?UTF-8?q?kspace?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit #807 Export, import, compare, and verify prompts and assistants across LangSmith environments Co-Authored-By: Claude Opus 5.5 --- .../export-import-prompts-and-assistants/design.md | 4 ++-- .../specs/prompt/spec.md | 9 ++++++++- .../export-import-prompts-and-assistants/tasks.md | 4 ++-- 3 files changed, 12 insertions(+), 5 deletions(-) diff --git a/openspec/changes/export-import-prompts-and-assistants/design.md b/openspec/changes/export-import-prompts-and-assistants/design.md index 629fb49d0..8fa135e0c 100644 --- a/openspec/changes/export-import-prompts-and-assistants/design.md +++ b/openspec/changes/export-import-prompts-and-assistants/design.md @@ -81,7 +81,7 @@ Without `--update`, import lists the differences and exits with status 3. `--dry ### Compare reports differences by JSON path -Compare reads the prompt or assistant from the destination, puts it in the shape of the export file's `repo`, `commit` and `assistant` objects, and lists every value that differs from the file. Compare checks every key in those objects, including the chat model's settings and the `id` arrays inside a manifest that name the prompt and model classes, such as `ChatOpenAI`. It skips `commit.model_config` and `commit.model_provider`, because LangSmith fills both in itself when import pushes a commit, and a destination workspace can fill them in differently from the source. Export leaves the values the server sets, such as IDs, version numbers, timestamps, owner and like counts, out of those objects, so they never show up as differences. The source environment's workspace, deployment and IDs sit in `source`, so they never show up as differences against a copy. Each difference names its JSON Pointer path and both values, except at a chat model setting whose name ends in `api_key`: there the report prints `` in place of a string on either side, so a key someone typed into the destination prompt's model never reaches a terminal or a CI log. For a prompt, the comparison covers the commit description, the commit tags and the commit each one points at, any tag on the compared commit that the file lacks, and the repo's description, readme, tags and visibility as well as the manifest. Import runs the same comparison after it writes, so a field LangSmith or the Agent Server drops shows up as a difference. +Compare reads the prompt or assistant from the destination, puts it in the shape of the export file's `repo`, `commit` and `assistant` objects, and lists every value that differs from the file. Compare checks every key in those objects, including the chat model's settings and the `id` arrays inside a manifest that name the prompt and model classes, such as `ChatOpenAI`. It skips `commit.model_config` and `commit.model_provider`, because LangSmith fills both in itself when import pushes a commit, and a destination workspace can fill them in differently from the source. Export leaves the values the server sets, such as IDs, version numbers, timestamps, owner and like counts, out of those objects, so they never show up as differences. The source environment's workspace, deployment and IDs sit in `source`, so they never show up as differences against a copy. Each difference names its JSON Pointer path and both values. Before the report prints a value, it puts `` in place of every string under a chat model setting whose name ends in `api_key`, whether that setting is at the difference's path or inside the value. This keeps a key someone typed into the destination prompt's model out of the terminal and the CI log. For a prompt, the comparison covers the commit description, the commit tags and the commit each one points at, any tag on the compared commit that the file lacks, and the repo's description, readme, tags and visibility as well as the manifest. Import runs the same comparison after it writes, so a field LangSmith or the Agent Server drops shows up as a difference. ### Verify checks what the copy needs to run @@ -107,7 +107,7 @@ Status 0 means the resource matches and every check passed. Status 3 means langs ### A Python script checks that an imported structured prompt still returns structured data -The LangSmith API has no documented way to run a prompt with its model, so a short Python script runs each imported structured prompt with LangChain and checks that the result matches the prompt's schema. The one prompt endpoint in the OpenAPI spec, `POST /api/v1/prompts/invoke_prompt`, takes message templates and inputs with no model or schema, and no LangSmith SDK calls it; in [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8), a developer calls it on the live API in case it does more than the spec says. A mise task runs the script with `uv run --script`, which installs the `langsmith` and `langchain-openai` packages the script lists, on a developer's machine and in the Integration Tests CI job. The script reads the destination workspace: it builds its LangSmith `Client` with `workspace_id` set to `LANGSTAR_TEST_DEST_WORKSPACE_ID`, because CI sets `LANGSMITH_WORKSPACE_ID` to the source workspace, which holds a prompt with the same handle. The script pulls the imported prompt with `include_model=True` and passes `secrets={"OPENAI_API_KEY": }` to [`pull_prompt`](https://github.com/langchain-ai/langsmith-sdk/blob/47c553caf08c1989e2be3ff1886da3eb801f709f/python/langsmith/client.py#L10228), because the LangSmith Python SDK reads no secret from the environment unless asked, and raises when a manifest holds a secret reference and no `secrets` map arrives. The script needs an OpenAI API key. codekiln creates it and stores it in 1Password, in the repository's CI secrets and as the destination workspace's `OPENAI_API_KEY` secret, and in [export-import-prompts-and-assistants Task 2.1 Add the destination workspace and the model provider key](tasks.md#L16) a developer adds it to `fnox.toml` and the CI workflow. +The LangSmith API has no documented way to run a prompt with its model, so a short Python script runs each imported structured prompt with LangChain and checks that the result matches the prompt's schema. The one prompt endpoint in the OpenAPI spec, `POST /api/v1/prompts/invoke_prompt`, takes message templates and inputs with no model or schema, and no LangSmith SDK calls it; in [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8), a developer calls it on the live API in case it does more than the spec says. A mise task runs the script with `uv run --script`, which installs the `langsmith` and `langchain-openai` packages the script lists, on a developer's machine and in the Integration Tests CI job. The script reads the destination workspace: it builds its LangSmith `Client` with `workspace_id` set to `LANGSTAR_TEST_DEST_WORKSPACE_ID`, because both `fnox.toml` and CI set `LANGSMITH_WORKSPACE_ID` to the source workspace, which holds a prompt with the same handle. The script pulls the imported prompt with `include_model=True` and passes `secrets={"OPENAI_API_KEY": }` to [`pull_prompt`](https://github.com/langchain-ai/langsmith-sdk/blob/47c553caf08c1989e2be3ff1886da3eb801f709f/python/langsmith/client.py#L10228), because the LangSmith Python SDK reads no secret from the environment unless asked, and raises when a manifest holds a secret reference and no `secrets` map arrives. The script needs an OpenAI API key. codekiln creates it and stores it in 1Password, in the repository's CI secrets and as the destination workspace's `OPENAI_API_KEY` secret, and in [export-import-prompts-and-assistants Task 2.1 Add the destination workspace and the model provider key](tasks.md#L16) a developer adds it to `fnox.toml` and the CI workflow. ### Export and import use the same SDK calls as the planned `prompt create`, `get` and `update` diff --git a/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md index 1876f6ea9..3ad35a1d3 100644 --- a/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md +++ b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md @@ -251,7 +251,7 @@ After writing, `langstar prompt import` SHALL record the commit hash in the dest ### Requirement: Redact an API key written into a destination prompt's model -When `prompt compare`, `prompt verify` or `prompt import` reports a difference at a chat model setting whose name ends in `api_key` and either side holds a string there, langstar SHALL print `` in place of that string in the text and JSON reports. +When `prompt compare`, `prompt verify` or `prompt import` prints the value on either side of a difference, langstar SHALL print `` in place of every string under a chat model setting whose name ends in `api_key`, whether the difference is at that setting or at an object that holds it, in the text and JSON reports. #### Scenario: Destination model holds a literal API key @@ -259,6 +259,13 @@ When `prompt compare`, `prompt verify` or `prompt import` reports a difference a - **AND** a user runs `langstar prompt compare my-prompt.json` - **THEN** langstar reports the setting's path with `` in place of the string and exits with status 3 +#### Scenario: Destination adds a model with a literal API key + +- **WHEN** the file holds a prompt saved without a model +- **AND** someone saved the destination prompt with a model whose `openai_api_key` is written as a string +- **AND** a user runs `langstar prompt compare my-prompt.json` +- **THEN** langstar reports the model's path with `` in place of the string and exits with status 3 + ### Requirement: Compare a prompt with its export file `langstar prompt compare ` SHALL read the prompt with the file's handle from the workspace, with its model and repo details, and list each difference by its JSON path with both values. `--commit ` SHALL choose the commit, and without it the latest. It SHALL compare every field in the file's `repo` and `commit`, including every manifest key and its `id` arrays, except `commit.model_config` and `commit.model_provider`, which import cannot set. diff --git a/openspec/changes/export-import-prompts-and-assistants/tasks.md b/openspec/changes/export-import-prompts-and-assistants/tasks.md index 888031c62..f5e44e9e2 100644 --- a/openspec/changes/export-import-prompts-and-assistants/tasks.md +++ b/openspec/changes/export-import-prompts-and-assistants/tasks.md @@ -40,7 +40,7 @@ Each integration test in this group carries `#[cfg_attr(not(feature = "integrati - Unit tests on nested objects, arrays, missing keys, and keys holding `/` or `~`, which the report escapes as RFC 6901 requires, so `a/b` appears as `/a~1b` and `a~b` as `/a~0b`. - A test that a changed `id` array inside `commit.manifest`, such as `ChatOpenAI` replaced by `ChatAnthropic`, is reported at its path. - A test that a destination whose `commit.model_config` and `commit.model_provider` differ from the file's, while the manifest matches, is reported as a match. -- [ ] 4.3 Add the shared result report, which `prompt compare`, `prompt verify` and `prompt import` all print through, and the exit statuses 0, 3 and 1. The report prints JSON to stdout and progress to stderr, and prints `` in place of a string at any chat model setting whose name ends in `api_key`, on either side of a difference. Leave clap's status 2 for usage errors. Verify with unit tests of the report, of the exit status each result gives, and of a difference at `openai_api_key` where the destination holds a string, asserting that the text and the `-f json` report each print `` and neither holds the string. +- [ ] 4.3 Add the shared result report, which the compare, verify and import commands for both prompts and assistants print through, and the exit statuses 0, 3 and 1. The report prints JSON to stdout and progress to stderr. Before it prints the value on either side of a difference, it puts `` in place of every string under a key whose name ends in `api_key`, anywhere inside that value. Leave clap's status 2 for usage errors. Verify with unit tests of the report, of the exit status each result gives, of a difference at `openai_api_key` where the destination holds a string, and of a difference at `/commit/manifest/kwargs/last` where the file has no model and the destination's model holds a string under `openai_api_key`. Each redaction test asserts that the text and the `-f json` report print `` and that neither holds the string. - [ ] 4.4 Add the secret reference walk that lists secret names from a manifest and returns the JSON Pointer path of each chat model setting whose name ends in `api_key` and holds a string instead of a secret reference, which `prompt export` from 5.1 refuses with status 1; verify with unit tests on a copy of the `prompt_with_model` manifest from the LangSmith Python SDK's integration tests, which `references/langsmith-sdk.md` links, on a manifest with no secret reference, and on a manifest whose `openai_api_key` holds a string, asserting that the walk returns that setting's path and that the error message holds the path and not the string ## 5. prompt export @@ -96,7 +96,7 @@ Each integration test in this group carries `#[cfg_attr(not(feature = "integrati - An integration test that `--update` against a destination whose latest commit matches the file and carries a `staging` tag the file lacks deletes `staging`, adds no commit and exits with status 0, and that `--dry-run` lists the delete. - An integration test that `--dry-run` writes nothing. - A unit test in `cli/src/commands/prompt.rs` that serves, through a `mockito` server and `LangchainClient::with_base_urls`, a commit read back that lacks a field the file holds, calls the import code, and asserts that the result names that field's JSON Pointer path and gives exit status 3, and that the destination record holds the new commit hash. -- [ ] 7.3 Add the structured output check as a mise task that runs a Python script with `uv run --script`. The script names `langsmith` and `langchain-openai` as inline script dependencies. It builds its `Client` with `workspace_id` set to `LANGSTAR_TEST_DEST_WORKSPACE_ID`, because CI sets `LANGSMITH_WORKSPACE_ID` to the source workspace, which holds a prompt with the same handle. It then pulls the imported prompt with `include_model=True` and a `secrets` map holding the model provider key read from the environment, invokes it and validates the result against the schema. Add `uv` to `mise.toml`, and remove `install: false` from the Setup mise step of the Integration Tests job in `.github/workflows/ci.yml` so that job installs `uv` and Python 3.11. Verify by having the import integration test create a `json_schema` prompt and a `function_calling` prompt, each saved with a small OpenAI model, import both into the destination workspace and run the check on each, under `mise run check` and in that CI job. +- [ ] 7.3 Add the structured output check as a mise task that runs a Python script with `uv run --script`. The script names `langsmith` and `langchain-openai` as inline script dependencies. It builds its `Client` with `workspace_id` set to `LANGSTAR_TEST_DEST_WORKSPACE_ID`, because both `fnox.toml` and CI set `LANGSMITH_WORKSPACE_ID` to the source workspace, which holds a prompt with the same handle. It then pulls the imported prompt with `include_model=True` and a `secrets` map holding the model provider key read from the environment, invokes it and validates the result against the schema. Add `uv` to `mise.toml`, and remove `install: false` from the Setup mise step of the Integration Tests job in `.github/workflows/ci.yml` so that job installs `uv` and Python 3.11. Verify by having the import integration test create a `json_schema` prompt and a `function_calling` prompt, each saved with a small OpenAI model, import both into the destination workspace and run the check on each, under `mise run check` and in that CI job. - [ ] 7.4 Document prompt import, the workspace scope of prompts and setting up destination credentials; verify that each documented command runs as written ## 8. assistant export From a5b921e2ad8819902dc90fd8bb13ed2b2761c8df Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Wed, 14 Oct 2026 03:43:30 -0400 Subject: [PATCH 61/90] =?UTF-8?q?=F0=9F=93=9A=20docs(openspec):=20redact?= =?UTF-8?q?=20any=20string=20a=20key=20ending=20in=20api=5Fkey=20holds,=20?= =?UTF-8?q?print=20secret=20references=20as=20they=20are,=20and=20redact?= =?UTF-8?q?=20assistant=20compare=20output=20too?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit #807 Export, import, compare, and verify prompts and assistants across LangSmith environments Co-Authored-By: Claude Opus 5.5 --- .../export-import-prompts-and-assistants/design.md | 2 +- .../specs/assistant/spec.md | 10 ++++++++++ .../specs/prompt/spec.md | 9 ++++++++- .../export-import-prompts-and-assistants/tasks.md | 2 +- 4 files changed, 20 insertions(+), 3 deletions(-) diff --git a/openspec/changes/export-import-prompts-and-assistants/design.md b/openspec/changes/export-import-prompts-and-assistants/design.md index 8fa135e0c..198199f0f 100644 --- a/openspec/changes/export-import-prompts-and-assistants/design.md +++ b/openspec/changes/export-import-prompts-and-assistants/design.md @@ -81,7 +81,7 @@ Without `--update`, import lists the differences and exits with status 3. `--dry ### Compare reports differences by JSON path -Compare reads the prompt or assistant from the destination, puts it in the shape of the export file's `repo`, `commit` and `assistant` objects, and lists every value that differs from the file. Compare checks every key in those objects, including the chat model's settings and the `id` arrays inside a manifest that name the prompt and model classes, such as `ChatOpenAI`. It skips `commit.model_config` and `commit.model_provider`, because LangSmith fills both in itself when import pushes a commit, and a destination workspace can fill them in differently from the source. Export leaves the values the server sets, such as IDs, version numbers, timestamps, owner and like counts, out of those objects, so they never show up as differences. The source environment's workspace, deployment and IDs sit in `source`, so they never show up as differences against a copy. Each difference names its JSON Pointer path and both values. Before the report prints a value, it puts `` in place of every string under a chat model setting whose name ends in `api_key`, whether that setting is at the difference's path or inside the value. This keeps a key someone typed into the destination prompt's model out of the terminal and the CI log. For a prompt, the comparison covers the commit description, the commit tags and the commit each one points at, any tag on the compared commit that the file lacks, and the repo's description, readme, tags and visibility as well as the manifest. Import runs the same comparison after it writes, so a field LangSmith or the Agent Server drops shows up as a difference. +Compare reads the prompt or assistant from the destination, puts it in the shape of the export file's `repo`, `commit` and `assistant` objects, and lists every value that differs from the file. Compare checks every key in those objects, including the chat model's settings and the `id` arrays inside a manifest that name the prompt and model classes, such as `ChatOpenAI`. It skips `commit.model_config` and `commit.model_provider`, because LangSmith fills both in itself when import pushes a commit, and a destination workspace can fill them in differently from the source. Export leaves the values the server sets, such as IDs, version numbers, timestamps, owner and like counts, out of those objects, so they never show up as differences. The source environment's workspace, deployment and IDs sit in `source`, so they never show up as differences against a copy. Each difference names its JSON Pointer path and both values. Before the report prints a value, it puts `` in place of each string that a key ending in `api_key` holds, whether that key is at the difference's path or inside the value, and prints a secret reference under such a key as it is. A key someone typed into the destination prompt's model therefore stays out of the terminal and the CI log, and the report still names the secret the model reads. For a prompt, the comparison covers the commit description, the commit tags and the commit each one points at, any tag on the compared commit that the file lacks, and the repo's description, readme, tags and visibility as well as the manifest. Import runs the same comparison after it writes, so a field LangSmith or the Agent Server drops shows up as a difference. ### Verify checks what the copy needs to run diff --git a/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md b/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md index d600ca447..b87ec2751 100644 --- a/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md +++ b/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md @@ -213,3 +213,13 @@ After writing, `langstar assistant import` SHALL record the assistant ID in the - **WHEN** a user runs `langstar -f json assistant verify support-bot.json --deployment my-agent-prod` - **THEN** stdout holds one JSON object with `status`, `differences` and `checks` + +### Requirement: Redact an API key written into a destination assistant's configuration + +When `assistant compare`, `assistant verify` or `assistant import` prints a value on either side of a difference, langstar SHALL print ``, in the text and JSON reports, in place of each string that a key ending in `api_key` holds as its value, whether that key sits at the difference's path or inside the value. + +#### Scenario: Destination configuration holds a literal API key + +- **WHEN** someone updated the destination assistant so `config.configurable.openai_api_key` holds a string +- **AND** a user runs `langstar assistant compare support-bot.json --deployment my-agent-prod` +- **THEN** langstar reports `/assistant/config/configurable/openai_api_key` with `` in place of the string and exits with status 3 diff --git a/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md index 3ad35a1d3..629df3f6a 100644 --- a/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md +++ b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md @@ -251,7 +251,7 @@ After writing, `langstar prompt import` SHALL record the commit hash in the dest ### Requirement: Redact an API key written into a destination prompt's model -When `prompt compare`, `prompt verify` or `prompt import` prints the value on either side of a difference, langstar SHALL print `` in place of every string under a chat model setting whose name ends in `api_key`, whether the difference is at that setting or at an object that holds it, in the text and JSON reports. +When `prompt compare`, `prompt verify` or `prompt import` prints a value on either side of a difference, langstar SHALL print ``, in the text and JSON reports, in place of each string that a key ending in `api_key` holds as its value, whether that key sits at the difference's path or inside the value. langstar SHALL print a secret reference under such a key as it is, with its secret name. #### Scenario: Destination model holds a literal API key @@ -266,6 +266,13 @@ When `prompt compare`, `prompt verify` or `prompt import` prints the value on ei - **AND** a user runs `langstar prompt compare my-prompt.json` - **THEN** langstar reports the model's path with `` in place of the string and exits with status 3 +#### Scenario: Destination adds a model that reads its key from a secret + +- **WHEN** the file holds a prompt saved without a model +- **AND** someone saved the destination prompt with a model whose `openai_api_key` refers to the secret `OPENAI_API_KEY` +- **AND** a user runs `langstar prompt compare my-prompt.json` +- **THEN** langstar reports the model's path with the secret reference and `OPENAI_API_KEY` in it, and exits with status 3 + ### Requirement: Compare a prompt with its export file `langstar prompt compare ` SHALL read the prompt with the file's handle from the workspace, with its model and repo details, and list each difference by its JSON path with both values. `--commit ` SHALL choose the commit, and without it the latest. It SHALL compare every field in the file's `repo` and `commit`, including every manifest key and its `id` arrays, except `commit.model_config` and `commit.model_provider`, which import cannot set. diff --git a/openspec/changes/export-import-prompts-and-assistants/tasks.md b/openspec/changes/export-import-prompts-and-assistants/tasks.md index f5e44e9e2..1d2b927a3 100644 --- a/openspec/changes/export-import-prompts-and-assistants/tasks.md +++ b/openspec/changes/export-import-prompts-and-assistants/tasks.md @@ -40,7 +40,7 @@ Each integration test in this group carries `#[cfg_attr(not(feature = "integrati - Unit tests on nested objects, arrays, missing keys, and keys holding `/` or `~`, which the report escapes as RFC 6901 requires, so `a/b` appears as `/a~1b` and `a~b` as `/a~0b`. - A test that a changed `id` array inside `commit.manifest`, such as `ChatOpenAI` replaced by `ChatAnthropic`, is reported at its path. - A test that a destination whose `commit.model_config` and `commit.model_provider` differ from the file's, while the manifest matches, is reported as a match. -- [ ] 4.3 Add the shared result report, which the compare, verify and import commands for both prompts and assistants print through, and the exit statuses 0, 3 and 1. The report prints JSON to stdout and progress to stderr. Before it prints the value on either side of a difference, it puts `` in place of every string under a key whose name ends in `api_key`, anywhere inside that value. Leave clap's status 2 for usage errors. Verify with unit tests of the report, of the exit status each result gives, of a difference at `openai_api_key` where the destination holds a string, and of a difference at `/commit/manifest/kwargs/last` where the file has no model and the destination's model holds a string under `openai_api_key`. Each redaction test asserts that the text and the `-f json` report print `` and that neither holds the string. +- [ ] 4.3 Add the shared result report, which the compare, verify and import commands for both prompts and assistants print through, and the exit statuses 0, 3 and 1. The report prints JSON to stdout and progress to stderr. Before it prints a value on either side of a difference, it puts `` in place of each string that a key ending in `api_key` holds as its value, whether that key is at the difference's path or inside the value, and prints a secret reference under such a key as it is. Leave clap's status 2 for usage errors. Verify with unit tests of the report, of the exit status each result gives, of a difference at `openai_api_key` where the destination holds a string, of a difference at `/commit/manifest/kwargs/last` where the file has no model and the destination's model holds a string under `openai_api_key`, and of a difference at `/assistant/config/configurable/openai_api_key` where the destination holds a string. Each redaction test asserts that the text and the `-f json` report print `` and that neither holds the string. Add a test of a difference at `/commit/manifest/kwargs/last` where the destination's model refers to the secret `OPENAI_API_KEY`, asserting that the report prints `OPENAI_API_KEY`. - [ ] 4.4 Add the secret reference walk that lists secret names from a manifest and returns the JSON Pointer path of each chat model setting whose name ends in `api_key` and holds a string instead of a secret reference, which `prompt export` from 5.1 refuses with status 1; verify with unit tests on a copy of the `prompt_with_model` manifest from the LangSmith Python SDK's integration tests, which `references/langsmith-sdk.md` links, on a manifest with no secret reference, and on a manifest whose `openai_api_key` holds a string, asserting that the walk returns that setting's path and that the error message holds the path and not the string ## 5. prompt export From efeb7708299416cd41ec1fb6c7a03b6f1873c963 Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Wed, 14 Oct 2026 03:52:15 -0400 Subject: [PATCH 62/90] =?UTF-8?q?=F0=9F=93=9A=20docs(openspec):=20refuse?= =?UTF-8?q?=20to=20export=20an=20assistant=20holding=20a=20literal=20API?= =?UTF-8?q?=20key,=20and=20match=20the=20api=5Fkey=20suffix=20without=20re?= =?UTF-8?q?gard=20to=20case?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit #807 Export, import, compare, and verify prompts and assistants across LangSmith environments Co-Authored-By: Claude Opus 5.5 --- .../design.md | 6 +++--- .../specs/assistant/spec.md | 18 +++++++++++++++++- .../specs/prompt/spec.md | 8 ++++++-- .../tasks.md | 11 ++++++----- 4 files changed, 32 insertions(+), 11 deletions(-) diff --git a/openspec/changes/export-import-prompts-and-assistants/design.md b/openspec/changes/export-import-prompts-and-assistants/design.md index 198199f0f..fe89a51b5 100644 --- a/openspec/changes/export-import-prompts-and-assistants/design.md +++ b/openspec/changes/export-import-prompts-and-assistants/design.md @@ -81,7 +81,7 @@ Without `--update`, import lists the differences and exits with status 3. `--dry ### Compare reports differences by JSON path -Compare reads the prompt or assistant from the destination, puts it in the shape of the export file's `repo`, `commit` and `assistant` objects, and lists every value that differs from the file. Compare checks every key in those objects, including the chat model's settings and the `id` arrays inside a manifest that name the prompt and model classes, such as `ChatOpenAI`. It skips `commit.model_config` and `commit.model_provider`, because LangSmith fills both in itself when import pushes a commit, and a destination workspace can fill them in differently from the source. Export leaves the values the server sets, such as IDs, version numbers, timestamps, owner and like counts, out of those objects, so they never show up as differences. The source environment's workspace, deployment and IDs sit in `source`, so they never show up as differences against a copy. Each difference names its JSON Pointer path and both values. Before the report prints a value, it puts `` in place of each string that a key ending in `api_key` holds, whether that key is at the difference's path or inside the value, and prints a secret reference under such a key as it is. A key someone typed into the destination prompt's model therefore stays out of the terminal and the CI log, and the report still names the secret the model reads. For a prompt, the comparison covers the commit description, the commit tags and the commit each one points at, any tag on the compared commit that the file lacks, and the repo's description, readme, tags and visibility as well as the manifest. Import runs the same comparison after it writes, so a field LangSmith or the Agent Server drops shows up as a difference. +Compare reads the prompt or assistant from the destination, puts it in the shape of the export file's `repo`, `commit` and `assistant` objects, and lists every value that differs from the file. Compare checks every key in those objects, including the chat model's settings and the `id` arrays inside a manifest that name the prompt and model classes, such as `ChatOpenAI`. It skips `commit.model_config` and `commit.model_provider`, because LangSmith fills both in itself when import pushes a commit, and a destination workspace can fill them in differently from the source. Export leaves the values the server sets, such as IDs, version numbers, timestamps, owner and like counts, out of those objects, so they never show up as differences. The source environment's workspace, deployment and IDs sit in `source`, so they never show up as differences against a copy. Each difference names its JSON Pointer path and both values. Before the report prints either side of a difference, it replaces every string stored under a setting whose name ends in `api_key`, compared without regard to case, with ``, whether the difference is at that setting or at an object that contains it. An API key someone pasted into the destination prompt's model or the destination assistant's configuration therefore stays out of the terminal and the CI log. A LangChain secret reference under that setting, `{"lc": 1, "type": "secret", "id": ["OPENAI_API_KEY"]}`, prints unchanged, so the report still names the secret the model reads. For a prompt, the comparison covers the commit description, the commit tags and the commit each one points at, any tag on the compared commit that the file lacks, and the repo's description, readme, tags and visibility as well as the manifest. Import runs the same comparison after it writes, so a field LangSmith or the Agent Server drops shows up as a difference. ### Verify checks what the copy needs to run @@ -118,8 +118,8 @@ The LangSmith API has no documented way to run a prompt with its model, so a sho - [LangSmith may store a pushed model differently from the source] → Import reads the prompt back and reports any difference, and in [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8) a developer checks the round trip against the live API before anyone builds the commands. - [Pushing a manifest identical to the latest commit may return 409] → Import compares first and pushes nothing when the prompt matches. - [The SDK changes for export and import add public fields to `Prompt`, `CommitManifestResponse`, `CommitData`, `CommitRequest`, `Assistant` and the assistant request structs, so a crate that builds one of those structs with a struct literal or destructures it with a pattern that names every field stops compiling] → Each pull request that adds a field updates every struct literal of that struct in this repository. [Should the SDK field additions count as a breaking change?](#9---should-the-sdk-field-additions-count-as-a-breaking-change) asks codekiln whether those pull requests also carry a `BREAKING CHANGE:` footer, which `docs/dev/git-scm-conventions.md` says makes the release a major version. -- [A prompt's manifest may hold a credential written as a literal model setting, rather than as a secret reference] → Export refuses with status 1 and writes nothing when a chat model setting whose name ends in `api_key` holds a string, and names that setting's path without its value. A credential under any other name, such as a header, still reaches the file, because langstar cannot tell it from any other string. The prompt export docs in [export-import-prompts-and-assistants Task 5.2 Document prompt export](tasks.md#L56) tell users to keep credentials in the workspace's secrets and to read an export file before committing it. -- [An assistant's configuration, context or metadata may hold a credential, which export copies into a file meant for Git] → Export writes those objects as the Agent Server returns them, since a credential there looks like any other string. The assistant export docs in [export-import-prompts-and-assistants Task 8.2 Document assistant export](tasks.md#L111) tell users to keep credentials in the deployment's environment variables or the workspace's secrets, and to read an export file before committing it. +- [A prompt's manifest may hold a credential written as a literal model setting, rather than as a secret reference] → Export refuses with status 1 and writes nothing when a chat model setting whose name ends in `api_key`, compared without regard to case, holds a string, and names that setting's path without its value. A credential under any other name, such as a header, still reaches the file, because langstar cannot tell it from any other string. The prompt export docs in [export-import-prompts-and-assistants Task 5.2 Document prompt export](tasks.md#L56) tell users to keep credentials in the workspace's secrets and to read an export file before committing it. +- [An assistant's configuration, context or metadata may hold a credential, which export copies into a file meant for Git] → Export refuses with status 1 and writes nothing when a string sits under a key whose name ends in `api_key`, compared without regard to case, anywhere in those objects, and names that key's path without its value. A credential under any other name still reaches the file, because langstar cannot tell it from any other string. The assistant export docs in [export-import-prompts-and-assistants Task 8.2 Document assistant export](tasks.md#L112) tell users to keep credentials in the deployment's environment variables or the workspace's secrets, and to read an export file before committing it. - [A tag in the destination may already point at another commit] → `--update` moves the tag, and `--dry-run` lists the move. - [`--update` deletes a tag on the matching destination commit that the file lacks, and a graph on a deployment may read that tag] → `--dry-run` lists each delete, and `assistant verify` reports a prompt tag an assistant refers to that the workspace no longer has. - [The control plane may refuse a second deployment from this repository with 409] → In [export-import-prompts-and-assistants Task 2.3 Register the test graph under a second graph ID](tasks.md#L18), a developer deploys the new `langgraph.json` once, before anyone builds the fixture. If the control plane answers with 409, the developer brings [Which second workspace and second deployment should the tests use?](#5---which-second-workspace-and-second-deployment-should-the-tests-use) back to codekiln. diff --git a/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md b/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md index b87ec2751..3a0190a97 100644 --- a/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md +++ b/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md @@ -21,6 +21,16 @@ - **THEN** stdout holds only the export file's JSON document - **AND** any progress message goes to stderr +### Requirement: Refuse to export an API key written into an assistant + +When the assistant's configuration, context or metadata holds a string under a key whose name ends in `api_key`, compared without regard to case, such as `openai_api_key` or `OPENAI_API_KEY`, `langstar assistant export` SHALL write no file and nothing to stdout, name that key's JSON Pointer path without the string, and exit with status 1. + +#### Scenario: Configuration holds a literal API key + +- **WHEN** the assistant's `config.configurable.OPENAI_API_KEY` holds a string +- **AND** a user runs `langstar assistant export --deployment my-agent --output support-bot.json` +- **THEN** langstar writes no file, names `/config/configurable/OPENAI_API_KEY` without the string and exits with status 1 + ### Requirement: Record the prompts an assistant refers to `langstar assistant export` SHALL accept `--prompt-ref `, repeatable, naming a string in the assistant's configuration or context that holds a prompt name. The file SHALL list each pointer and the prompt name it holds under `requires.prompts`. A pointer that resolves to no string SHALL be an error. @@ -216,10 +226,16 @@ After writing, `langstar assistant import` SHALL record the assistant ID in the ### Requirement: Redact an API key written into a destination assistant's configuration -When `assistant compare`, `assistant verify` or `assistant import` prints a value on either side of a difference, langstar SHALL print ``, in the text and JSON reports, in place of each string that a key ending in `api_key` holds as its value, whether that key sits at the difference's path or inside the value. +When `assistant compare`, `assistant verify` or `assistant import` prints either side of a difference, in the text or the JSON report, langstar SHALL print `` in place of every string stored under a key whose name ends in `api_key`, compared without regard to case, so `openai_api_key` and `OPENAI_API_KEY` both match. This holds when the difference is at that key and when the key sits deeper inside the printed object. #### Scenario: Destination configuration holds a literal API key - **WHEN** someone updated the destination assistant so `config.configurable.openai_api_key` holds a string - **AND** a user runs `langstar assistant compare support-bot.json --deployment my-agent-prod` - **THEN** langstar reports `/assistant/config/configurable/openai_api_key` with `` in place of the string and exits with status 3 + +#### Scenario: Destination configuration holds a literal API key under an upper-case name + +- **WHEN** someone updated the destination assistant so `config.configurable.OPENAI_API_KEY` holds a string +- **AND** a user runs `langstar assistant compare support-bot.json --deployment my-agent-prod` +- **THEN** langstar reports `/assistant/config/configurable/OPENAI_API_KEY` with `` in place of the string and exits with status 3 diff --git a/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md index 629df3f6a..cfa0599e4 100644 --- a/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md +++ b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md @@ -68,7 +68,7 @@ The export file SHALL list, under `requires.secrets`, the name of every secret t ### Requirement: Refuse to export an API key written into a prompt's model -When a chat model constructor in the manifest holds a string, rather than a secret reference, under a setting whose name ends in `api_key`, such as `openai_api_key`, `langstar prompt export` SHALL write no file and nothing to stdout, name that setting's JSON Pointer path without its value, and exit with status 1. +When a chat model constructor in the manifest holds a string, rather than a secret reference, under a setting whose name ends in `api_key`, compared without regard to case, such as `openai_api_key` or `OPENAI_API_KEY`, `langstar prompt export` SHALL write no file and nothing to stdout, name that setting's JSON Pointer path without its value, and exit with status 1. #### Scenario: Model holds a literal API key @@ -251,7 +251,7 @@ After writing, `langstar prompt import` SHALL record the commit hash in the dest ### Requirement: Redact an API key written into a destination prompt's model -When `prompt compare`, `prompt verify` or `prompt import` prints a value on either side of a difference, langstar SHALL print ``, in the text and JSON reports, in place of each string that a key ending in `api_key` holds as its value, whether that key sits at the difference's path or inside the value. langstar SHALL print a secret reference under such a key as it is, with its secret name. +When `prompt compare`, `prompt verify` or `prompt import` prints either side of a difference, in the text or the JSON report, langstar SHALL print `` in place of every string stored under a key whose name ends in `api_key`, compared without regard to case. This holds when the difference is at that key and when the key sits deeper inside the printed object. #### Scenario: Destination model holds a literal API key @@ -266,6 +266,10 @@ When `prompt compare`, `prompt verify` or `prompt import` prints a value on eith - **AND** a user runs `langstar prompt compare my-prompt.json` - **THEN** langstar reports the model's path with `` in place of the string and exits with status 3 +### Requirement: Print a secret reference in a destination prompt's model unchanged + +When `prompt compare`, `prompt verify` or `prompt import` prints either side of a difference, in the text or the JSON report, and a key whose name ends in `api_key`, compared without regard to case, holds a LangChain secret reference, such as `{"lc": 1, "type": "secret", "id": ["OPENAI_API_KEY"]}`, langstar SHALL print the reference unchanged, so the report names the secret. + #### Scenario: Destination adds a model that reads its key from a secret - **WHEN** the file holds a prompt saved without a model diff --git a/openspec/changes/export-import-prompts-and-assistants/tasks.md b/openspec/changes/export-import-prompts-and-assistants/tasks.md index 1d2b927a3..1253fb3a1 100644 --- a/openspec/changes/export-import-prompts-and-assistants/tasks.md +++ b/openspec/changes/export-import-prompts-and-assistants/tasks.md @@ -40,8 +40,8 @@ Each integration test in this group carries `#[cfg_attr(not(feature = "integrati - Unit tests on nested objects, arrays, missing keys, and keys holding `/` or `~`, which the report escapes as RFC 6901 requires, so `a/b` appears as `/a~1b` and `a~b` as `/a~0b`. - A test that a changed `id` array inside `commit.manifest`, such as `ChatOpenAI` replaced by `ChatAnthropic`, is reported at its path. - A test that a destination whose `commit.model_config` and `commit.model_provider` differ from the file's, while the manifest matches, is reported as a match. -- [ ] 4.3 Add the shared result report, which the compare, verify and import commands for both prompts and assistants print through, and the exit statuses 0, 3 and 1. The report prints JSON to stdout and progress to stderr. Before it prints a value on either side of a difference, it puts `` in place of each string that a key ending in `api_key` holds as its value, whether that key is at the difference's path or inside the value, and prints a secret reference under such a key as it is. Leave clap's status 2 for usage errors. Verify with unit tests of the report, of the exit status each result gives, of a difference at `openai_api_key` where the destination holds a string, of a difference at `/commit/manifest/kwargs/last` where the file has no model and the destination's model holds a string under `openai_api_key`, and of a difference at `/assistant/config/configurable/openai_api_key` where the destination holds a string. Each redaction test asserts that the text and the `-f json` report print `` and that neither holds the string. Add a test of a difference at `/commit/manifest/kwargs/last` where the destination's model refers to the secret `OPENAI_API_KEY`, asserting that the report prints `OPENAI_API_KEY`. -- [ ] 4.4 Add the secret reference walk that lists secret names from a manifest and returns the JSON Pointer path of each chat model setting whose name ends in `api_key` and holds a string instead of a secret reference, which `prompt export` from 5.1 refuses with status 1; verify with unit tests on a copy of the `prompt_with_model` manifest from the LangSmith Python SDK's integration tests, which `references/langsmith-sdk.md` links, on a manifest with no secret reference, and on a manifest whose `openai_api_key` holds a string, asserting that the walk returns that setting's path and that the error message holds the path and not the string +- [ ] 4.3 Add the shared result report, which the compare, verify and import commands for both prompts and assistants print through, and the exit statuses 0, 3 and 1. The report prints JSON to stdout and progress to stderr. Before it prints either side of a difference, it puts `` in place of every string stored under a key whose name ends in `api_key`, compared without regard to case, whether the difference is at that key or at an object that contains it, and prints a secret reference under that key unchanged. Leave clap's status 2 for usage errors. Verify with unit tests of the report, of the exit status each result gives, of a difference at `openai_api_key` where the destination holds a string, of a difference at `/commit/manifest/kwargs/last` where the file has no model and the destination's model holds a string under `openai_api_key`, of a difference at `/assistant/config/configurable/openai_api_key` where the destination holds a string, and of a difference at `/assistant/config/configurable/OPENAI_API_KEY` where the destination holds a string. Each redaction test asserts that the text and the `-f json` report print `` and that neither holds the string. Add a test of a difference at `/commit/manifest/kwargs/last` where the destination's model refers to the secret `OPENAI_API_KEY`, asserting that the text and the `-f json` report both print the secret reference with `OPENAI_API_KEY` in it, and that neither prints ``. +- [ ] 4.4 Add the secret reference walk that lists secret names from a manifest and returns the JSON Pointer path of each chat model setting whose name ends in `api_key`, compared without regard to case, and holds a string instead of a secret reference, which `prompt export` from 5.1 refuses with status 1; verify with unit tests on a copy of the `prompt_with_model` manifest from the LangSmith Python SDK's integration tests, which `references/langsmith-sdk.md` links, on a manifest with no secret reference, and on manifests whose `openai_api_key` and whose `OPENAI_API_KEY` hold a string, asserting that the walk returns that setting's path and that the error message holds the path and not the string ## 5. prompt export @@ -51,7 +51,7 @@ Each integration test in this group carries `#[cfg_attr(not(feature = "integrati - An error test that exports a handle the workspace has no prompt for, and asserts that it exits with status 1, names the handle and writes no file. - A CLI test that runs `prompt export` without `--output`, parses all of stdout as one JSON document, and asserts that its `source.commit_hash` matches the SDK's read and that any progress message is on stderr. - An integration test that gives one commit two tags, exports it twice, and asserts that the two files are identical. - - A unit test in `cli/src/commands/prompt.rs` that serves, through a `mockito` server and `LangchainClient::with_base_urls`, a commit whose `ChatOpenAI` constructor holds a string under `openai_api_key`, calls the export code with `--output my-prompt.json`, and asserts that it returns an error that exits with status 1 and names the setting's path without the string, and that it writes no file. + - A unit test in `cli/src/commands/prompt.rs` that serves, through a `mockito` server and `LangchainClient::with_base_urls`, a commit whose `ChatOpenAI` constructor holds a string under `openai_api_key`, and another under `OPENAI_API_KEY`, calls the export code on each with `--output my-prompt.json`, and asserts that it returns an error that exits with status 1 and names the setting's path without the string, and that it writes no file. - A unit test in `cli/src/commands/prompt.rs` that serves a prompt through a `mockito` server and `LangchainClient::with_base_urls`, calls the export code with no workspace ID set, and asserts that `source.workspace_id` holds the repo's `tenant_id`. - [ ] 5.2 Document `prompt export` in `docs/usage/` and the README. The page says that export refuses a prompt whose model holds an API key as a literal string, names that setting and writes no file, and that a credential kept under any other name, such as a header, still reaches the file; tells users to keep credentials in the workspace's secrets; and tells them to read an export file before they commit it. Verify that each documented command runs as written. @@ -101,14 +101,15 @@ Each integration test in this group carries `#[cfg_attr(not(feature = "integrati ## 8. assistant export -- [ ] 8.1 Add `langstar assistant export` with `--output` and `--prompt-ref`. Verify with these tests: +- [ ] 8.1 Add `langstar assistant export` with `--output`, `--prompt-ref` and the refusal of a string under a key whose name ends in `api_key`, compared without regard to case, anywhere in the configuration, context or metadata. Verify with these tests: - An integration test that creates an assistant with configuration, context and metadata through the SDK and asserts every field of the exported file. - An integration test that exports, with `--prompt-ref /config/configurable/system_prompt`, an assistant whose value there is `support-system:prod`, and asserts that the file holds that pointer and `support-system:prod` under `requires.prompts`. - An integration test that exports the same assistant version twice and asserts that the two files are identical. - An error test for a pointer that names nothing. - An error test that exporting an assistant ID the deployment lacks exits with status 1, names the ID and writes no file. + - A unit test in `cli/src/commands/assistant.rs` that serves, through a `mockito` server and `LangchainClient::with_base_urls`, an assistant whose `config.configurable.OPENAI_API_KEY` holds a string, and another whose `metadata.openai_api_key` holds a string, calls the export code on each with `--output support-bot.json`, and asserts that it returns an error that exits with status 1 and names the key's JSON Pointer path without the string, and that it writes no file. - A CLI test that runs `assistant export` without `--output`, parses all of stdout as one JSON document, and asserts that its `source.assistant_id` is the exported assistant's ID and that any progress message is on stderr. -- [ ] 8.2 Document assistant export. The page says that export writes the assistant's configuration, context and metadata as the Agent Server returns them, so a credential kept there reaches the file; tells users to keep credentials in the deployment's environment variables or the workspace's secrets; and tells them to read an export file before they commit it. Verify that each documented command runs as written. +- [ ] 8.2 Document assistant export. The page says that export refuses an assistant whose configuration, context or metadata holds a string under a key whose name ends in `api_key`, names that key's path and writes no file, and that a credential kept under any other name still reaches the file; tells users to keep credentials in the deployment's environment variables or the workspace's secrets; and tells them to read an export file before they commit it. Verify that each documented command runs as written. ## 9. assistant compare and verify From 7152017e62643a9ae768c15143b8f4eb18353961 Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Wed, 14 Oct 2026 04:00:51 -0400 Subject: [PATCH 63/90] =?UTF-8?q?=F0=9F=93=9A=20docs(openspec):=20redact?= =?UTF-8?q?=20only=20a=20string=20value=20under=20an=20api=5Fkey=20key,=20?= =?UTF-8?q?print=20an=20assistant's=20secret=20reference=20unchanged,=20an?= =?UTF-8?q?d=20test=20the=20context=20branch=20of=20the=20assistant=20expo?= =?UTF-8?q?rt=20refusal?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit #807 Export, import, compare, and verify prompts and assistants across LangSmith environments Co-Authored-By: Claude Opus 5.5 --- .../export-import-prompts-and-assistants/design.md | 2 +- .../specs/assistant/spec.md | 12 +++++++++++- .../specs/prompt/spec.md | 2 +- .../export-import-prompts-and-assistants/tasks.md | 4 ++-- 4 files changed, 15 insertions(+), 5 deletions(-) diff --git a/openspec/changes/export-import-prompts-and-assistants/design.md b/openspec/changes/export-import-prompts-and-assistants/design.md index fe89a51b5..358c1dec3 100644 --- a/openspec/changes/export-import-prompts-and-assistants/design.md +++ b/openspec/changes/export-import-prompts-and-assistants/design.md @@ -81,7 +81,7 @@ Without `--update`, import lists the differences and exits with status 3. `--dry ### Compare reports differences by JSON path -Compare reads the prompt or assistant from the destination, puts it in the shape of the export file's `repo`, `commit` and `assistant` objects, and lists every value that differs from the file. Compare checks every key in those objects, including the chat model's settings and the `id` arrays inside a manifest that name the prompt and model classes, such as `ChatOpenAI`. It skips `commit.model_config` and `commit.model_provider`, because LangSmith fills both in itself when import pushes a commit, and a destination workspace can fill them in differently from the source. Export leaves the values the server sets, such as IDs, version numbers, timestamps, owner and like counts, out of those objects, so they never show up as differences. The source environment's workspace, deployment and IDs sit in `source`, so they never show up as differences against a copy. Each difference names its JSON Pointer path and both values. Before the report prints either side of a difference, it replaces every string stored under a setting whose name ends in `api_key`, compared without regard to case, with ``, whether the difference is at that setting or at an object that contains it. An API key someone pasted into the destination prompt's model or the destination assistant's configuration therefore stays out of the terminal and the CI log. A LangChain secret reference under that setting, `{"lc": 1, "type": "secret", "id": ["OPENAI_API_KEY"]}`, prints unchanged, so the report still names the secret the model reads. For a prompt, the comparison covers the commit description, the commit tags and the commit each one points at, any tag on the compared commit that the file lacks, and the repo's description, readme, tags and visibility as well as the manifest. Import runs the same comparison after it writes, so a field LangSmith or the Agent Server drops shows up as a difference. +Compare reads the prompt or assistant from the destination, puts it in the shape of the export file's `repo`, `commit` and `assistant` objects, and lists every value that differs from the file. Compare checks every key in those objects, including the chat model's settings and the `id` arrays inside a manifest that name the prompt and model classes, such as `ChatOpenAI`. It skips `commit.model_config` and `commit.model_provider`, because LangSmith fills both in itself when import pushes a commit, and a destination workspace can fill them in differently from the source. Export leaves the values the server sets, such as IDs, version numbers, timestamps, owner and like counts, out of those objects, so they never show up as differences. The source environment's workspace, deployment and IDs sit in `source`, so they never show up as differences against a copy. Each difference names its JSON Pointer path and both values. Before the report prints either side of a difference, it replaces with `` the string value of every setting whose name ends in `api_key`, compared without regard to case, whether the difference is at that setting or at an object that contains it. An API key someone pasted into the destination prompt's model or the destination assistant's configuration therefore stays out of the terminal and the CI log. When the setting holds a LangChain secret reference, `{"lc": 1, "type": "secret", "id": ["OPENAI_API_KEY"]}`, the report prints the reference unchanged, so the reader still sees which secret the model reads. For a prompt, the comparison covers the commit description, the commit tags and the commit each one points at, any tag on the compared commit that the file lacks, and the repo's description, readme, tags and visibility as well as the manifest. Import runs the same comparison after it writes, so a field LangSmith or the Agent Server drops shows up as a difference. ### Verify checks what the copy needs to run diff --git a/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md b/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md index 3a0190a97..66043e5db 100644 --- a/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md +++ b/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md @@ -226,7 +226,7 @@ After writing, `langstar assistant import` SHALL record the assistant ID in the ### Requirement: Redact an API key written into a destination assistant's configuration -When `assistant compare`, `assistant verify` or `assistant import` prints either side of a difference, in the text or the JSON report, langstar SHALL print `` in place of every string stored under a key whose name ends in `api_key`, compared without regard to case, so `openai_api_key` and `OPENAI_API_KEY` both match. This holds when the difference is at that key and when the key sits deeper inside the printed object. +When `assistant compare`, `assistant verify` or `assistant import` prints either side of a difference, in the text or the JSON report, langstar SHALL print `` wherever a key whose name ends in `api_key`, compared without regard to case, such as `openai_api_key` or `OPENAI_API_KEY`, has a string as its value, whether the difference is at that key or at an object that contains it. #### Scenario: Destination configuration holds a literal API key @@ -239,3 +239,13 @@ When `assistant compare`, `assistant verify` or `assistant import` prints either - **WHEN** someone updated the destination assistant so `config.configurable.OPENAI_API_KEY` holds a string - **AND** a user runs `langstar assistant compare support-bot.json --deployment my-agent-prod` - **THEN** langstar reports `/assistant/config/configurable/OPENAI_API_KEY` with `` in place of the string and exits with status 3 + +### Requirement: Print a secret reference in a destination assistant's configuration unchanged + +When `assistant compare`, `assistant verify` or `assistant import` prints either side of a difference, in the text or the JSON report, and a key whose name ends in `api_key`, compared without regard to case, holds a LangChain secret reference, such as `{"lc": 1, "type": "secret", "id": ["OPENAI_API_KEY"]}`, langstar SHALL print the reference unchanged, so the report names the secret. + +#### Scenario: Destination configuration reads its key from a secret + +- **WHEN** someone updated the destination assistant so `config.configurable.openai_api_key` refers to the secret `OPENAI_API_KEY` +- **AND** a user runs `langstar assistant compare support-bot.json --deployment my-agent-prod` +- **THEN** langstar reports `/assistant/config/configurable/openai_api_key` with the secret reference and `OPENAI_API_KEY` in it, and exits with status 3 diff --git a/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md index cfa0599e4..f85b6205b 100644 --- a/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md +++ b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md @@ -251,7 +251,7 @@ After writing, `langstar prompt import` SHALL record the commit hash in the dest ### Requirement: Redact an API key written into a destination prompt's model -When `prompt compare`, `prompt verify` or `prompt import` prints either side of a difference, in the text or the JSON report, langstar SHALL print `` in place of every string stored under a key whose name ends in `api_key`, compared without regard to case. This holds when the difference is at that key and when the key sits deeper inside the printed object. +When `prompt compare`, `prompt verify` or `prompt import` prints either side of a difference, in the text or the JSON report, langstar SHALL print `` wherever a key whose name ends in `api_key`, compared without regard to case, has a string as its value, whether the difference is at that key or at an object that contains it. #### Scenario: Destination model holds a literal API key diff --git a/openspec/changes/export-import-prompts-and-assistants/tasks.md b/openspec/changes/export-import-prompts-and-assistants/tasks.md index 1253fb3a1..9b0c9cb78 100644 --- a/openspec/changes/export-import-prompts-and-assistants/tasks.md +++ b/openspec/changes/export-import-prompts-and-assistants/tasks.md @@ -40,7 +40,7 @@ Each integration test in this group carries `#[cfg_attr(not(feature = "integrati - Unit tests on nested objects, arrays, missing keys, and keys holding `/` or `~`, which the report escapes as RFC 6901 requires, so `a/b` appears as `/a~1b` and `a~b` as `/a~0b`. - A test that a changed `id` array inside `commit.manifest`, such as `ChatOpenAI` replaced by `ChatAnthropic`, is reported at its path. - A test that a destination whose `commit.model_config` and `commit.model_provider` differ from the file's, while the manifest matches, is reported as a match. -- [ ] 4.3 Add the shared result report, which the compare, verify and import commands for both prompts and assistants print through, and the exit statuses 0, 3 and 1. The report prints JSON to stdout and progress to stderr. Before it prints either side of a difference, it puts `` in place of every string stored under a key whose name ends in `api_key`, compared without regard to case, whether the difference is at that key or at an object that contains it, and prints a secret reference under that key unchanged. Leave clap's status 2 for usage errors. Verify with unit tests of the report, of the exit status each result gives, of a difference at `openai_api_key` where the destination holds a string, of a difference at `/commit/manifest/kwargs/last` where the file has no model and the destination's model holds a string under `openai_api_key`, of a difference at `/assistant/config/configurable/openai_api_key` where the destination holds a string, and of a difference at `/assistant/config/configurable/OPENAI_API_KEY` where the destination holds a string. Each redaction test asserts that the text and the `-f json` report print `` and that neither holds the string. Add a test of a difference at `/commit/manifest/kwargs/last` where the destination's model refers to the secret `OPENAI_API_KEY`, asserting that the text and the `-f json` report both print the secret reference with `OPENAI_API_KEY` in it, and that neither prints ``. +- [ ] 4.3 Add the shared result report, which the compare, verify and import commands for both prompts and assistants print through, and the exit statuses 0, 3 and 1. The report prints JSON to stdout and progress to stderr. Before it prints either side of a difference, it puts `` wherever a key whose name ends in `api_key`, compared without regard to case, has a string as its value, whether the difference is at that key or at an object that contains it, and prints a secret reference under that key unchanged. Leave clap's status 2 for usage errors. Verify with unit tests of the report, of the exit status each result gives, of a difference at `openai_api_key` where the destination holds a string, of a difference at `/commit/manifest/kwargs/last` where the file has no model and the destination's model holds a string under `openai_api_key`, of a difference at `/assistant/config/configurable/openai_api_key` where the destination holds a string, and of a difference at `/assistant/config/configurable/OPENAI_API_KEY` where the destination holds a string. Each redaction test asserts that the text and the `-f json` report print `` and that neither holds the string. Add a test of a difference at `/commit/manifest/kwargs/last` where the destination's model refers to the secret `OPENAI_API_KEY`, asserting that the text and the `-f json` report both print the secret reference with `OPENAI_API_KEY` in it, and that neither prints ``. Add a test of a difference at `/assistant/config/configurable/openai_api_key` where the destination refers to the secret `OPENAI_API_KEY`, asserting that the text and the `-f json` report both print the secret reference with `OPENAI_API_KEY` in it, and that neither prints ``. - [ ] 4.4 Add the secret reference walk that lists secret names from a manifest and returns the JSON Pointer path of each chat model setting whose name ends in `api_key`, compared without regard to case, and holds a string instead of a secret reference, which `prompt export` from 5.1 refuses with status 1; verify with unit tests on a copy of the `prompt_with_model` manifest from the LangSmith Python SDK's integration tests, which `references/langsmith-sdk.md` links, on a manifest with no secret reference, and on manifests whose `openai_api_key` and whose `OPENAI_API_KEY` hold a string, asserting that the walk returns that setting's path and that the error message holds the path and not the string ## 5. prompt export @@ -107,7 +107,7 @@ Each integration test in this group carries `#[cfg_attr(not(feature = "integrati - An integration test that exports the same assistant version twice and asserts that the two files are identical. - An error test for a pointer that names nothing. - An error test that exporting an assistant ID the deployment lacks exits with status 1, names the ID and writes no file. - - A unit test in `cli/src/commands/assistant.rs` that serves, through a `mockito` server and `LangchainClient::with_base_urls`, an assistant whose `config.configurable.OPENAI_API_KEY` holds a string, and another whose `metadata.openai_api_key` holds a string, calls the export code on each with `--output support-bot.json`, and asserts that it returns an error that exits with status 1 and names the key's JSON Pointer path without the string, and that it writes no file. + - A unit test in `cli/src/commands/assistant.rs` that serves, through a `mockito` server and `LangchainClient::with_base_urls`, three assistants: one whose `config.configurable.OPENAI_API_KEY` holds a string, one whose `context.anthropic_api_key` holds a string, and one whose `metadata.openai_api_key` holds a string. It calls the export code on each with `--output support-bot.json` and asserts that each call returns an error that exits with status 1 and names the key's JSON Pointer path without the string, and that it writes no file. - A CLI test that runs `assistant export` without `--output`, parses all of stdout as one JSON document, and asserts that its `source.assistant_id` is the exported assistant's ID and that any progress message is on stderr. - [ ] 8.2 Document assistant export. The page says that export refuses an assistant whose configuration, context or metadata holds a string under a key whose name ends in `api_key`, names that key's path and writes no file, and that a credential kept under any other name still reaches the file; tells users to keep credentials in the deployment's environment variables or the workspace's secrets; and tells them to read an export file before they commit it. Verify that each documented command runs as written. From 055f9b191e724432798f68c341f8e347e2813429 Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Wed, 14 Oct 2026 04:08:35 -0400 Subject: [PATCH 64/90] =?UTF-8?q?=F0=9F=93=9A=20docs(openspec):=20say=20wh?= =?UTF-8?q?at=20``=20replaces,=20lead=20the=20assistant=20secret?= =?UTF-8?q?=20reference=20requirement=20with=20its=20SHALL,=20and=20justif?= =?UTF-8?q?y=20printing=20a=20secret=20reference=20by=20what=20it=20holds?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit #807 Export, import, compare, and verify prompts and assistants across LangSmith environments Co-Authored-By: Claude Opus 5.5 --- .../changes/export-import-prompts-and-assistants/design.md | 2 +- .../specs/assistant/spec.md | 4 ++-- .../export-import-prompts-and-assistants/specs/prompt/spec.md | 2 +- .../changes/export-import-prompts-and-assistants/tasks.md | 2 +- 4 files changed, 5 insertions(+), 5 deletions(-) diff --git a/openspec/changes/export-import-prompts-and-assistants/design.md b/openspec/changes/export-import-prompts-and-assistants/design.md index 358c1dec3..171226152 100644 --- a/openspec/changes/export-import-prompts-and-assistants/design.md +++ b/openspec/changes/export-import-prompts-and-assistants/design.md @@ -81,7 +81,7 @@ Without `--update`, import lists the differences and exits with status 3. `--dry ### Compare reports differences by JSON path -Compare reads the prompt or assistant from the destination, puts it in the shape of the export file's `repo`, `commit` and `assistant` objects, and lists every value that differs from the file. Compare checks every key in those objects, including the chat model's settings and the `id` arrays inside a manifest that name the prompt and model classes, such as `ChatOpenAI`. It skips `commit.model_config` and `commit.model_provider`, because LangSmith fills both in itself when import pushes a commit, and a destination workspace can fill them in differently from the source. Export leaves the values the server sets, such as IDs, version numbers, timestamps, owner and like counts, out of those objects, so they never show up as differences. The source environment's workspace, deployment and IDs sit in `source`, so they never show up as differences against a copy. Each difference names its JSON Pointer path and both values. Before the report prints either side of a difference, it replaces with `` the string value of every setting whose name ends in `api_key`, compared without regard to case, whether the difference is at that setting or at an object that contains it. An API key someone pasted into the destination prompt's model or the destination assistant's configuration therefore stays out of the terminal and the CI log. When the setting holds a LangChain secret reference, `{"lc": 1, "type": "secret", "id": ["OPENAI_API_KEY"]}`, the report prints the reference unchanged, so the reader still sees which secret the model reads. For a prompt, the comparison covers the commit description, the commit tags and the commit each one points at, any tag on the compared commit that the file lacks, and the repo's description, readme, tags and visibility as well as the manifest. Import runs the same comparison after it writes, so a field LangSmith or the Agent Server drops shows up as a difference. +Compare reads the prompt or assistant from the destination, puts it in the shape of the export file's `repo`, `commit` and `assistant` objects, and lists every value that differs from the file. Compare checks every key in those objects, including the chat model's settings and the `id` arrays inside a manifest that name the prompt and model classes, such as `ChatOpenAI`. It skips `commit.model_config` and `commit.model_provider`, because LangSmith fills both in itself when import pushes a commit, and a destination workspace can fill them in differently from the source. Export leaves the values the server sets, such as IDs, version numbers, timestamps, owner and like counts, out of those objects, so they never show up as differences. The source environment's workspace, deployment and IDs sit in `source`, so they never show up as differences against a copy. Each difference names its JSON Pointer path and both values. Before the report prints either side of a difference, it replaces with `` the string value of every setting whose name ends in `api_key`, compared without regard to case, whether the difference is at that setting or at an object that contains it. An API key someone pasted into the destination prompt's model or the destination assistant's configuration therefore stays out of the terminal and the CI log. When the setting holds a LangChain secret reference, such as `{"lc": 1, "type": "secret", "id": ["OPENAI_API_KEY"]}`, the report prints it unchanged, because the reference holds only the secret's name, never its value. For a prompt, the comparison covers the commit description, the commit tags and the commit each one points at, any tag on the compared commit that the file lacks, and the repo's description, readme, tags and visibility as well as the manifest. Import runs the same comparison after it writes, so a field LangSmith or the Agent Server drops shows up as a difference. ### Verify checks what the copy needs to run diff --git a/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md b/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md index 66043e5db..3bc3ffb55 100644 --- a/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md +++ b/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md @@ -226,7 +226,7 @@ After writing, `langstar assistant import` SHALL record the assistant ID in the ### Requirement: Redact an API key written into a destination assistant's configuration -When `assistant compare`, `assistant verify` or `assistant import` prints either side of a difference, in the text or the JSON report, langstar SHALL print `` wherever a key whose name ends in `api_key`, compared without regard to case, such as `openai_api_key` or `OPENAI_API_KEY`, has a string as its value, whether the difference is at that key or at an object that contains it. +When `assistant compare`, `assistant verify` or `assistant import` prints either side of a difference, in the text or the JSON report, langstar SHALL print `` in place of the value of each key whose name ends in `api_key` when that value is a string. langstar matches the name without regard to case, so `openai_api_key` and `OPENAI_API_KEY` both match. This holds when the difference is at that key and when it is at an object that contains the key. #### Scenario: Destination configuration holds a literal API key @@ -242,7 +242,7 @@ When `assistant compare`, `assistant verify` or `assistant import` prints either ### Requirement: Print a secret reference in a destination assistant's configuration unchanged -When `assistant compare`, `assistant verify` or `assistant import` prints either side of a difference, in the text or the JSON report, and a key whose name ends in `api_key`, compared without regard to case, holds a LangChain secret reference, such as `{"lc": 1, "type": "secret", "id": ["OPENAI_API_KEY"]}`, langstar SHALL print the reference unchanged, so the report names the secret. +When `assistant compare`, `assistant verify` or `assistant import` prints either side of a difference, in the text or the JSON report, langstar SHALL print a LangChain secret reference unchanged when it is the value of a key whose name ends in `api_key`, so the report names the secret. A secret reference looks like `{"lc": 1, "type": "secret", "id": ["OPENAI_API_KEY"]}`, and langstar matches the key name without regard to case. #### Scenario: Destination configuration reads its key from a secret diff --git a/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md index f85b6205b..fd4a55f5f 100644 --- a/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md +++ b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md @@ -251,7 +251,7 @@ After writing, `langstar prompt import` SHALL record the commit hash in the dest ### Requirement: Redact an API key written into a destination prompt's model -When `prompt compare`, `prompt verify` or `prompt import` prints either side of a difference, in the text or the JSON report, langstar SHALL print `` wherever a key whose name ends in `api_key`, compared without regard to case, has a string as its value, whether the difference is at that key or at an object that contains it. +When `prompt compare`, `prompt verify` or `prompt import` prints either side of a difference, in the text or the JSON report, langstar SHALL print `` in place of the value of each key whose name ends in `api_key` when that value is a string. langstar matches the name without regard to case. This holds when the difference is at that key and when it is at an object that contains the key. #### Scenario: Destination model holds a literal API key diff --git a/openspec/changes/export-import-prompts-and-assistants/tasks.md b/openspec/changes/export-import-prompts-and-assistants/tasks.md index 9b0c9cb78..6ed514222 100644 --- a/openspec/changes/export-import-prompts-and-assistants/tasks.md +++ b/openspec/changes/export-import-prompts-and-assistants/tasks.md @@ -40,7 +40,7 @@ Each integration test in this group carries `#[cfg_attr(not(feature = "integrati - Unit tests on nested objects, arrays, missing keys, and keys holding `/` or `~`, which the report escapes as RFC 6901 requires, so `a/b` appears as `/a~1b` and `a~b` as `/a~0b`. - A test that a changed `id` array inside `commit.manifest`, such as `ChatOpenAI` replaced by `ChatAnthropic`, is reported at its path. - A test that a destination whose `commit.model_config` and `commit.model_provider` differ from the file's, while the manifest matches, is reported as a match. -- [ ] 4.3 Add the shared result report, which the compare, verify and import commands for both prompts and assistants print through, and the exit statuses 0, 3 and 1. The report prints JSON to stdout and progress to stderr. Before it prints either side of a difference, it puts `` wherever a key whose name ends in `api_key`, compared without regard to case, has a string as its value, whether the difference is at that key or at an object that contains it, and prints a secret reference under that key unchanged. Leave clap's status 2 for usage errors. Verify with unit tests of the report, of the exit status each result gives, of a difference at `openai_api_key` where the destination holds a string, of a difference at `/commit/manifest/kwargs/last` where the file has no model and the destination's model holds a string under `openai_api_key`, of a difference at `/assistant/config/configurable/openai_api_key` where the destination holds a string, and of a difference at `/assistant/config/configurable/OPENAI_API_KEY` where the destination holds a string. Each redaction test asserts that the text and the `-f json` report print `` and that neither holds the string. Add a test of a difference at `/commit/manifest/kwargs/last` where the destination's model refers to the secret `OPENAI_API_KEY`, asserting that the text and the `-f json` report both print the secret reference with `OPENAI_API_KEY` in it, and that neither prints ``. Add a test of a difference at `/assistant/config/configurable/openai_api_key` where the destination refers to the secret `OPENAI_API_KEY`, asserting that the text and the `-f json` report both print the secret reference with `OPENAI_API_KEY` in it, and that neither prints ``. +- [ ] 4.3 Add the shared result report, which the compare, verify and import commands for both prompts and assistants print through, and the exit statuses 0, 3 and 1. The report prints JSON to stdout and progress to stderr. Before it prints either side of a difference, the report puts `` in place of the value of each key whose name ends in `api_key`, matched without regard to case, when that value is a string, and prints a secret reference under such a key unchanged. It does this both when the difference is at the key and when it is at an object that contains the key. Leave clap's status 2 for usage errors. Verify with unit tests of the report, of the exit status each result gives, of a difference at `openai_api_key` where the destination holds a string, of a difference at `/commit/manifest/kwargs/last` where the file has no model and the destination's model holds a string under `openai_api_key`, of a difference at `/assistant/config/configurable/openai_api_key` where the destination holds a string, and of a difference at `/assistant/config/configurable/OPENAI_API_KEY` where the destination holds a string. Each redaction test asserts that the text and the `-f json` report print `` and that neither holds the string. Add a test of a difference at `/commit/manifest/kwargs/last` where the destination's model refers to the secret `OPENAI_API_KEY`, asserting that the text and the `-f json` report both print the secret reference with `OPENAI_API_KEY` in it, and that neither prints ``. Add a test of a difference at `/assistant/config/configurable/openai_api_key` where the destination refers to the secret `OPENAI_API_KEY`, asserting that the text and the `-f json` report both print the secret reference with `OPENAI_API_KEY` in it, and that neither prints ``. - [ ] 4.4 Add the secret reference walk that lists secret names from a manifest and returns the JSON Pointer path of each chat model setting whose name ends in `api_key`, compared without regard to case, and holds a string instead of a secret reference, which `prompt export` from 5.1 refuses with status 1; verify with unit tests on a copy of the `prompt_with_model` manifest from the LangSmith Python SDK's integration tests, which `references/langsmith-sdk.md` links, on a manifest with no secret reference, and on manifests whose `openai_api_key` and whose `OPENAI_API_KEY` hold a string, asserting that the walk returns that setting's path and that the error message holds the path and not the string ## 5. prompt export From 532b9f5116f5f27397b10d1365a8fda8de157c3b Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Wed, 14 Oct 2026 04:17:01 -0400 Subject: [PATCH 65/90] =?UTF-8?q?=F0=9F=93=9A=20docs(openspec):=20lead=20t?= =?UTF-8?q?he=20prompt=20secret=20reference=20requirement=20with=20its=20S?= =?UTF-8?q?HALL,=20as=20the=20assistant=20one=20does?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit #807 Export, import, compare, and verify prompts and assistants across LangSmith environments Co-Authored-By: Claude Opus 5.5 --- .../export-import-prompts-and-assistants/specs/prompt/spec.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md index fd4a55f5f..d4e2e2221 100644 --- a/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md +++ b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md @@ -268,7 +268,7 @@ When `prompt compare`, `prompt verify` or `prompt import` prints either side of ### Requirement: Print a secret reference in a destination prompt's model unchanged -When `prompt compare`, `prompt verify` or `prompt import` prints either side of a difference, in the text or the JSON report, and a key whose name ends in `api_key`, compared without regard to case, holds a LangChain secret reference, such as `{"lc": 1, "type": "secret", "id": ["OPENAI_API_KEY"]}`, langstar SHALL print the reference unchanged, so the report names the secret. +When `prompt compare`, `prompt verify` or `prompt import` prints either side of a difference, in the text or the JSON report, langstar SHALL print a LangChain secret reference unchanged when it is the value of a key whose name ends in `api_key`, so the report names the secret. A secret reference looks like `{"lc": 1, "type": "secret", "id": ["OPENAI_API_KEY"]}`, and langstar matches the key name without regard to case. #### Scenario: Destination adds a model that reads its key from a secret From 4b2f75ba20b5cd641a0541cb8451a243e412e16b Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Wed, 14 Oct 2026 04:20:30 -0400 Subject: [PATCH 66/90] =?UTF-8?q?=F0=9F=93=9A=20docs(openspec):=20state=20?= =?UTF-8?q?the=20redaction=20case=20match=20and=20its=20reach=20as=20SHALL?= =?UTF-8?q?s=20in=20both=20specs?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit #807 Export, import, compare, and verify prompts and assistants across LangSmith environments Co-Authored-By: Claude Opus 5.5 --- .../specs/assistant/spec.md | 2 +- .../export-import-prompts-and-assistants/specs/prompt/spec.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md b/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md index 3bc3ffb55..0fc77a593 100644 --- a/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md +++ b/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md @@ -226,7 +226,7 @@ After writing, `langstar assistant import` SHALL record the assistant ID in the ### Requirement: Redact an API key written into a destination assistant's configuration -When `assistant compare`, `assistant verify` or `assistant import` prints either side of a difference, in the text or the JSON report, langstar SHALL print `` in place of the value of each key whose name ends in `api_key` when that value is a string. langstar matches the name without regard to case, so `openai_api_key` and `OPENAI_API_KEY` both match. This holds when the difference is at that key and when it is at an object that contains the key. +When `assistant compare`, `assistant verify` or `assistant import` prints either side of a difference, in the text or the JSON report, langstar SHALL print `` in place of the value of each key whose name ends in `api_key` when that value is a string. langstar SHALL match the name without regard to case. langstar SHALL redact the string both when the difference is at the key and when the difference is at an object that contains the key. #### Scenario: Destination configuration holds a literal API key diff --git a/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md index d4e2e2221..165cb060d 100644 --- a/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md +++ b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md @@ -251,7 +251,7 @@ After writing, `langstar prompt import` SHALL record the commit hash in the dest ### Requirement: Redact an API key written into a destination prompt's model -When `prompt compare`, `prompt verify` or `prompt import` prints either side of a difference, in the text or the JSON report, langstar SHALL print `` in place of the value of each key whose name ends in `api_key` when that value is a string. langstar matches the name without regard to case. This holds when the difference is at that key and when it is at an object that contains the key. +When `prompt compare`, `prompt verify` or `prompt import` prints either side of a difference, in the text or the JSON report, langstar SHALL print `` in place of the value of each key whose name ends in `api_key` when that value is a string. langstar SHALL match the name without regard to case. langstar SHALL redact the string both when the difference is at the key and when the difference is at an object that contains the key. #### Scenario: Destination model holds a literal API key From b3e8407d13e5e36e06a79b4e7fd132f22f06b8e6 Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Tue, 27 Oct 2026 16:23:37 -0400 Subject: [PATCH 67/90] =?UTF-8?q?=F0=9F=93=9A=20docs(openspec):=20refuse?= =?UTF-8?q?=20an=20export=20that=20holds=20any=20credential,=20found=20by?= =?UTF-8?q?=20key=20name=20or=20value=20shape,=20and=20redact=20the=20same?= =?UTF-8?q?=20strings=20in=20reports?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Export now checks every string it would write, not only settings ending in api_key: key names with secret, token, password, credential, authorization or cookie, and values shaped like bearer tokens, private keys, URLs and connection strings with passwords, JSON Web Tokens and provider API keys. It refuses with status 1 instead of redacting, and --not-a-secret lets one reviewed string through. Co-Authored-By: Claude Opus 5.5 --- .../design.md | 24 ++++- .../proposal.md | 1 + .../specs/assistant/spec.md | 77 ++++++++++++++-- .../specs/prompt/spec.md | 88 +++++++++++++++++-- .../tasks.md | 29 ++++-- 5 files changed, 197 insertions(+), 22 deletions(-) diff --git a/openspec/changes/export-import-prompts-and-assistants/design.md b/openspec/changes/export-import-prompts-and-assistants/design.md index 171226152..ad6cb7268 100644 --- a/openspec/changes/export-import-prompts-and-assistants/design.md +++ b/openspec/changes/export-import-prompts-and-assistants/design.md @@ -55,6 +55,21 @@ Without `--output`, export prints the file to stdout and nothing else, so `langs Import, compare and verify read a file only when its `format_version` is 1 and its `kind` matches the command: `prompt` for the prompt commands and `assistant` for the assistant commands. For any other file the command writes nothing, exits with status 1 and names the field and the value it found. When a user runs `prompt import` on an assistant export, or on a file a newer langstar wrote, langstar stops with that error before it changes anything in the destination. +### Export refuses to write a file that holds a credential + +An export file is meant for Git, and a secret pushed to Git stays in its history after anyone deletes it. So before it writes anything, export runs a credential check over every string it would write in the `repo`, `commit` and `assistant` objects. For a prompt that covers the manifest with each model setting, header, `model_kwargs` entry, message template and `metadata` value in it, `commit.model_config`, the commit description and the repo's description and readme. For an assistant it covers the name, description, configuration, context and metadata. The check skips a LangChain secret reference, `{"lc": 1, "type": "secret", "id": ["OPENAI_API_KEY"]}`, because it holds only a secret's name, and skips the empty string. + +The check flags a string by two rules. + +- **The key's name.** Langstar splits the name of the key that holds the string, or that holds the array the string is in, into words at `_`, `-`, `.`, spaces and each change from a lower-case to an upper-case letter, and compares the words without regard to case. A name with one of the words `secret`, `token`, `password`, `passwd`, `pwd`, `credential`, `credentials`, `apikey`, `authorization` or `cookie`, or with the word `key` right after `api`, `access`, `private`, `secret`, `signing` or `account`, is flagged. So `openai_api_key`, `OPENAI_API_KEY`, `x-api-key`, `Authorization`, `access_token`, `client_secret`, `db_password` and `privateKey` all are, and `max_tokens` and `cache_key` are not. +- **The value's shape**, under any key. Langstar flags a string that holds a bearer token (`Bearer ` followed by at least 16 token characters) or a `Basic ` value followed by at least 16 base64 characters; a PEM private key (`-----BEGIN` and `PRIVATE KEY-----`); a URL with a password in it, `scheme://user:password@host` or `scheme://:password@host`, which covers database and queue connection strings; a key-value connection string with a non-empty `Password=`, `Pwd=`, `AccountKey=`, `SharedAccessKey=` or `SharedAccessSignature=`; a JSON Web Token, three base64url parts that start with `eyJ`; or a token with a provider's published prefix, which are `sk-` (OpenAI and Anthropic), `lsv2_` (LangSmith), `ghp_`, `gho_`, `ghs_` and `github_pat_` (GitHub), `xoxb-`, `xoxp-` and `xoxa-` (Slack), `AKIA` and `ASIA` followed by 16 upper-case letters or digits (AWS), `AIza` (Google) and `hf_` (Hugging Face), each followed by enough characters to rule out an ordinary word. + +When the check flags any string, export writes no file and nothing to stdout, lists the JSON Pointer path of each flagged string with the rule that flagged it, such as `key name` or `URL with a password`, never the string or any part of it, and exits with status 1. Export never redacts a credential and writes the rest. A redacted file would import a prompt or assistant that cannot authenticate, and a reviewer reading the file in a pull request would not learn that the source holds a credential. The refusal asks the user to move the credential into the workspace's secrets, which a prompt's model reads through a secret reference, or into the deployment's environment variables, which a graph reads. + +A string the check flags may hold no secret, such as a message template that shows `Bearer ` followed by an example token. `--not-a-secret `, which a user can give more than once, tells export to write the string at that one path, and export prints a warning to stderr naming each such path. A pointer at a string the check does not flag makes export write nothing, name the pointer and exit with status 1, so a script cannot keep a stale exemption. No flag turns the check off. + +The check cannot recognize a credential that has none of these names or shapes, such as a random string under a key called `x`. The export docs tell users to keep credentials in secrets and environment variables, and to read an export file before committing it. + ### Export asks LangSmith for the model Export reads the commit with `include=model`, the form the Go SDK uses (`references/langsmith-go.md`). The Python SDK sends the older `include_model=true`, and the LangSmith OpenAPI spec accepts both. With the model included, the manifest wraps the prompt and the chat model with its settings in one constructor, and the response also carries `model_config` and `model_provider` beside the manifest (`references/langsmith-sdk.md`). No field of the create-commit request holds the model, so the model reaches the destination inside the manifest, and export keeps `model_config` and `model_provider` in the file only for the reader. The LangSmith Python SDK's [`PromptCommit`](https://github.com/langchain-ai/langsmith-sdk/blob/47c553caf08c1989e2be3ff1886da3eb801f709f/python/langsmith/schemas.py#L1054) documents `model_provider` as the model's class name, such as `ChatOpenAI`. In [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8), a developer pushes that manifest to a second workspace and records whether LangSmith returns the same manifest, `model_config` and `model_provider`, and which outer `id` the live API returns. @@ -81,7 +96,7 @@ Without `--update`, import lists the differences and exits with status 3. `--dry ### Compare reports differences by JSON path -Compare reads the prompt or assistant from the destination, puts it in the shape of the export file's `repo`, `commit` and `assistant` objects, and lists every value that differs from the file. Compare checks every key in those objects, including the chat model's settings and the `id` arrays inside a manifest that name the prompt and model classes, such as `ChatOpenAI`. It skips `commit.model_config` and `commit.model_provider`, because LangSmith fills both in itself when import pushes a commit, and a destination workspace can fill them in differently from the source. Export leaves the values the server sets, such as IDs, version numbers, timestamps, owner and like counts, out of those objects, so they never show up as differences. The source environment's workspace, deployment and IDs sit in `source`, so they never show up as differences against a copy. Each difference names its JSON Pointer path and both values. Before the report prints either side of a difference, it replaces with `` the string value of every setting whose name ends in `api_key`, compared without regard to case, whether the difference is at that setting or at an object that contains it. An API key someone pasted into the destination prompt's model or the destination assistant's configuration therefore stays out of the terminal and the CI log. When the setting holds a LangChain secret reference, such as `{"lc": 1, "type": "secret", "id": ["OPENAI_API_KEY"]}`, the report prints it unchanged, because the reference holds only the secret's name, never its value. For a prompt, the comparison covers the commit description, the commit tags and the commit each one points at, any tag on the compared commit that the file lacks, and the repo's description, readme, tags and visibility as well as the manifest. Import runs the same comparison after it writes, so a field LangSmith or the Agent Server drops shows up as a difference. +Compare reads the prompt or assistant from the destination, puts it in the shape of the export file's `repo`, `commit` and `assistant` objects, and lists every value that differs from the file. Compare checks every key in those objects, including the chat model's settings and the `id` arrays inside a manifest that name the prompt and model classes, such as `ChatOpenAI`. It skips `commit.model_config` and `commit.model_provider`, because LangSmith fills both in itself when import pushes a commit, and a destination workspace can fill them in differently from the source. Export leaves the values the server sets, such as IDs, version numbers, timestamps, owner and like counts, out of those objects, so they never show up as differences. The source environment's workspace, deployment and IDs sit in `source`, so they never show up as differences against a copy. Each difference names its JSON Pointer path and both values. Before the report prints either side of a difference, it replaces with `` every string that export's credential check flags, by either rule, whether the difference is at that string or at an object that contains it. A credential someone pasted into the destination prompt or the destination assistant, or into the file, therefore stays out of the terminal and the CI log. The report prints a LangChain secret reference, such as `{"lc": 1, "type": "secret", "id": ["OPENAI_API_KEY"]}`, unchanged, because the reference holds only the secret's name, never its value. The report redacts rather than refuses because it writes to a terminal or log, not to a file meant for Git, and the user still needs to see where the difference is. For a prompt, the comparison covers the commit description, the commit tags and the commit each one points at, any tag on the compared commit that the file lacks, and the repo's description, readme, tags and visibility as well as the manifest. Import runs the same comparison after it writes, so a field LangSmith or the Agent Server drops shows up as a difference. ### Verify checks what the copy needs to run @@ -118,8 +133,9 @@ The LangSmith API has no documented way to run a prompt with its model, so a sho - [LangSmith may store a pushed model differently from the source] → Import reads the prompt back and reports any difference, and in [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8) a developer checks the round trip against the live API before anyone builds the commands. - [Pushing a manifest identical to the latest commit may return 409] → Import compares first and pushes nothing when the prompt matches. - [The SDK changes for export and import add public fields to `Prompt`, `CommitManifestResponse`, `CommitData`, `CommitRequest`, `Assistant` and the assistant request structs, so a crate that builds one of those structs with a struct literal or destructures it with a pattern that names every field stops compiling] → Each pull request that adds a field updates every struct literal of that struct in this repository. [Should the SDK field additions count as a breaking change?](#9---should-the-sdk-field-additions-count-as-a-breaking-change) asks codekiln whether those pull requests also carry a `BREAKING CHANGE:` footer, which `docs/dev/git-scm-conventions.md` says makes the release a major version. -- [A prompt's manifest may hold a credential written as a literal model setting, rather than as a secret reference] → Export refuses with status 1 and writes nothing when a chat model setting whose name ends in `api_key`, compared without regard to case, holds a string, and names that setting's path without its value. A credential under any other name, such as a header, still reaches the file, because langstar cannot tell it from any other string. The prompt export docs in [export-import-prompts-and-assistants Task 5.2 Document prompt export](tasks.md#L56) tell users to keep credentials in the workspace's secrets and to read an export file before committing it. -- [An assistant's configuration, context or metadata may hold a credential, which export copies into a file meant for Git] → Export refuses with status 1 and writes nothing when a string sits under a key whose name ends in `api_key`, compared without regard to case, anywhere in those objects, and names that key's path without its value. A credential under any other name still reaches the file, because langstar cannot tell it from any other string. The assistant export docs in [export-import-prompts-and-assistants Task 8.2 Document assistant export](tasks.md#L112) tell users to keep credentials in the deployment's environment variables or the workspace's secrets, and to read an export file before committing it. +- [A prompt or an assistant may hold a credential, such as an API key, a token, a password, an authorization header, a connection string with a password or a private key, which export would copy into a file meant for Git] → Export runs the credential check from [Export refuses to write a file that holds a credential](#export-refuses-to-write-a-file-that-holds-a-credential) and, when it flags a string, writes nothing, names each path and exits with status 1. Only `--not-a-secret ` lets a flagged string through, one path at a time. +- [The credential check misses a credential with no telling key name or value shape, such as a random string under a key called `x`] → The export docs in [export-import-prompts-and-assistants Task 5.2 Document prompt export](tasks.md#L67) and [export-import-prompts-and-assistants Task 8.2 Document assistant export](tasks.md#L125) list what the check catches and what it cannot, tell users to keep credentials in the workspace's secrets or the deployment's environment variables, and tell them to read an export file before committing it. +- [The check flags a string that holds no secret, such as an example token in a message template] → Export names the path and the rule, and the user passes `--not-a-secret` with that path after reading the string. - [A tag in the destination may already point at another commit] → `--update` moves the tag, and `--dry-run` lists the move. - [`--update` deletes a tag on the matching destination commit that the file lacks, and a graph on a deployment may read that tag] → `--dry-run` lists each delete, and `assistant verify` reports a prompt tag an assistant refers to that the workspace no longer has. - [The control plane may refuse a second deployment from this repository with 409] → In [export-import-prompts-and-assistants Task 2.3 Register the test graph under a second graph ID](tasks.md#L18), a developer deploys the new `langgraph.json` once, before anyone builds the fixture. If the control plane answers with 409, the developer brings [Which second workspace and second deployment should the tests use?](#5---which-second-workspace-and-second-deployment-should-the-tests-use) back to codekiln. @@ -147,7 +163,7 @@ None. ### 3 - How should the end-to-end test show that an imported structured prompt still returns structured data? > Context from the drafting agent, for question 3. -> The LangSmith OpenAPI spec's only prompt-invoke endpoint, `POST /api/v1/prompts/invoke_prompt`, takes message templates and inputs with no model or schema, and no SDK calls it. In [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8), a developer calls it on the live API. The draft runs a Python script through a mise task with `uv run --script`. The script pulls the prompt from the destination workspace with the LangSmith Python SDK's `pull_prompt`, runs it with LangChain, and names both packages as inline dependencies. [export-import-prompts-and-assistants Task 7.3 Add the structured output check](tasks.md#L99) adds `uv` to `mise.toml` and removes `install: false` from the Integration Tests CI job, so that job runs `mise install`. It needs a model provider key, such as `OPENAI_API_KEY`, which codekiln would create and store in 1Password, the CI secrets and the destination workspace's secrets. Building the model call into langstar as `prompt verify --invoke` would need langstar to call each provider's API itself. Recommendation: the Python script, with an OpenAI key and a small model. +> The LangSmith OpenAPI spec's only prompt-invoke endpoint, `POST /api/v1/prompts/invoke_prompt`, takes message templates and inputs with no model or schema, and no SDK calls it. In [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8), a developer calls it on the live API. The draft runs a Python script through a mise task with `uv run --script`. The script pulls the prompt from the destination workspace with the LangSmith Python SDK's `pull_prompt`, runs it with LangChain, and names both packages as inline dependencies. [export-import-prompts-and-assistants Task 7.3 Add the structured output check](tasks.md#L110) adds `uv` to `mise.toml` and removes `install: false` from the Integration Tests CI job, so that job runs `mise install`. It needs a model provider key, such as `OPENAI_API_KEY`, which codekiln would create and store in 1Password, the CI secrets and the destination workspace's secrets. Building the model call into langstar as `prompt verify --invoke` would need langstar to call each provider's API itself. Recommendation: the Python script, with an OpenAI key and a small model. diff --git a/openspec/changes/export-import-prompts-and-assistants/proposal.md b/openspec/changes/export-import-prompts-and-assistants/proposal.md index 770d3094b..2c43c17e4 100644 --- a/openspec/changes/export-import-prompts-and-assistants/proposal.md +++ b/openspec/changes/export-import-prompts-and-assistants/proposal.md @@ -14,6 +14,7 @@ A prompt belongs to a workspace and an assistant belongs to a deployment, but bo - `langstar prompt import --workspace-id ` creates the prompt in the destination workspace, or adds a commit to an existing prompt, updates its repo details and creates, moves or deletes its commit tags when the user passes `--update`. An import that stopped before it pushed the first commit finishes on the next run, and one that stopped before it created the commit tags finishes with `--update`. `--dry-run` shows what it would do. After writing, it reads the prompt back and reports any difference. - `langstar prompt compare ` reports the differences between the file and a prompt in a workspace. `langstar prompt verify ` also checks that the workspace has the secrets the prompt's model reads, and notes when the workspace has no saved model configuration for the model's class, such as `ChatOpenAI`. - `langstar assistant export`, `import`, `compare` and `verify` do the same for assistants on a deployment. The file holds the name, description, graph ID, configuration, context, metadata and any prompt names the assistant refers to. `verify` checks that the destination deployment has the graph and that the destination workspace has the referenced prompts. +- Both exports refuse to write a file that holds a credential. Before writing, export checks every string it would write for a key name such as `api_key`, `token`, `password`, `secret` or `Authorization`, and for a value shaped like a bearer token, a private key, a URL or connection string with a password, a JSON Web Token or a provider's API key. When one matches, export names each path, writes nothing and exits with status 1. `--not-a-secret ` lets one flagged string through after the user has read it. Compare, verify and import print `` in place of the same strings. - Import records each destination ID (the new commit hash, or the new assistant ID) in a separate file next to the export. When a user imports the same file again, langstar reads that file to find the assistant it created earlier instead of creating another one, and to tell whether someone pushed a commit to the prompt since the last import. - Import runs the dependency checks of `verify` before it writes, and writes nothing when one fails. - `compare`, `verify` and `import` share one exit status scheme, 0 for a match, 3 for a difference or a failed check and 1 for any other error, and print a JSON report with `-f json`. Status 2 stays clap's usage error. diff --git a/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md b/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md index 0fc77a593..60ac01f6d 100644 --- a/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md +++ b/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md @@ -21,9 +21,9 @@ - **THEN** stdout holds only the export file's JSON document - **AND** any progress message goes to stderr -### Requirement: Refuse to export an API key written into an assistant +### Requirement: Refuse to export an assistant that holds a credential -When the assistant's configuration, context or metadata holds a string under a key whose name ends in `api_key`, compared without regard to case, such as `openai_api_key` or `OPENAI_API_KEY`, `langstar assistant export` SHALL write no file and nothing to stdout, name that key's JSON Pointer path without the string, and exit with status 1. +Before it writes, `langstar assistant export` SHALL run the credential check over every string at any depth of the `assistant` object, skipping secret references and empty strings. When the check flags a string, export SHALL write no file and nothing to stdout, list the JSON Pointer path and rule of each flagged string, without the string or any part of it, and exit with status 1. Export SHALL NOT redact a flagged string and write the rest. #### Scenario: Configuration holds a literal API key @@ -31,6 +31,60 @@ When the assistant's configuration, context or metadata holds a string under a k - **AND** a user runs `langstar assistant export --deployment my-agent --output support-bot.json` - **THEN** langstar writes no file, names `/config/configurable/OPENAI_API_KEY` without the string and exits with status 1 +#### Scenario: Configuration reads its key from a secret + +- **WHEN** the assistant's `config.configurable.openai_api_key` holds a secret reference to `OPENAI_API_KEY` +- **AND** a user runs `langstar assistant export --deployment my-agent --output support-bot.json` +- **THEN** langstar writes the file with the secret reference and exits with status 0 + +### Requirement: Flag an assistant string by the name of its key + +The credential check SHALL flag a string when the key that holds it, or the array it is in, has a name whose words, split at `_`, `-`, `.`, spaces and lower-to-upper case changes and compared without regard to case, include `secret`, `token`, `password`, `passwd`, `pwd`, `credential`, `credentials`, `apikey`, `authorization` or `cookie`, or `key` right after `api`, `access`, `private`, `secret`, `signing` or `account`. + +#### Scenario: Context holds an access token + +- **WHEN** the assistant's `context.github_access_token` holds a string +- **AND** a user runs `langstar assistant export --deployment my-agent --output support-bot.json` +- **THEN** langstar writes no file, names `/context/github_access_token` without the string and exits with status 1 + +#### Scenario: Configuration holds a database password + +- **WHEN** the assistant's `config.configurable.db.password` holds a string +- **AND** a user runs `langstar assistant export --deployment my-agent --output support-bot.json` +- **THEN** langstar writes no file, names `/config/configurable/db/password` without the string and exits with status 1 + +#### Scenario: Configuration sends an authorization header + +- **WHEN** the assistant's `config.configurable.tool_headers.Authorization` holds `Bearer ` and a token +- **AND** a user runs `langstar assistant export --deployment my-agent --output support-bot.json` +- **THEN** langstar writes no file, names `/config/configurable/tool_headers/Authorization` without the string and exits with status 1 + +### Requirement: Flag an assistant string by its shape under any key + +The credential check SHALL flag a string, under any key, that holds a bearer token, a `Basic ` authorization value, a PEM private key, a URL with a password in its user info, a connection string with a non-empty `Password=`, `Pwd=`, `AccountKey=`, `SharedAccessKey=` or `SharedAccessSignature=`, a JSON Web Token, or a token that starts with a provider prefix the design lists: `sk-`, `lsv2_`, `ghp_`, `gho_`, `ghs_`, `github_pat_`, `xoxb-`, `xoxp-`, `xoxa-`, `AKIA`, `ASIA`, `AIza` or `hf_`. + +#### Scenario: Configuration holds a connection string with a password + +- **WHEN** the assistant's `config.configurable.checkpoint_uri` holds `postgresql://app:@db.example.com/agent` +- **AND** a user runs `langstar assistant export --deployment my-agent --output support-bot.json` +- **THEN** langstar writes no file, names `/config/configurable/checkpoint_uri` as a URL with a password and exits with status 1 + +#### Scenario: Metadata holds a private key + +- **WHEN** the assistant's `metadata.deploy_note` holds a PEM block that begins `-----BEGIN OPENSSH PRIVATE KEY-----` +- **AND** a user runs `langstar assistant export --deployment my-agent` +- **THEN** stdout is empty, langstar names `/metadata/deploy_note` as a private key and exits with status 1 + +### Requirement: Let the user export a flagged assistant string that holds no secret + +`langstar assistant export` SHALL take `--not-a-secret ` any number of times. For each pointer at a string the credential check flags, export SHALL write that string and print a warning to stderr naming the pointer. A flagged string at any path not named SHALL still make export refuse. When a pointer names a path the check does not flag, export SHALL write nothing, name the pointer and exit with status 1. No flag SHALL turn the check off. + +#### Scenario: Configuration names a token budget + +- **WHEN** the assistant's `config.configurable.token_budget_policy` holds the string `strict` +- **AND** a user runs `langstar assistant export --deployment my-agent --output support-bot.json --not-a-secret /config/configurable/token_budget_policy` +- **THEN** langstar writes the file with `strict` at that path, prints a warning naming the path to stderr and exits with status 0 + ### Requirement: Record the prompts an assistant refers to `langstar assistant export` SHALL accept `--prompt-ref `, repeatable, naming a string in the assistant's configuration or context that holds a prompt name. The file SHALL list each pointer and the prompt name it holds under `requires.prompts`. A pointer that resolves to no string SHALL be an error. @@ -224,9 +278,9 @@ After writing, `langstar assistant import` SHALL record the assistant ID in the - **WHEN** a user runs `langstar -f json assistant verify support-bot.json --deployment my-agent-prod` - **THEN** stdout holds one JSON object with `status`, `differences` and `checks` -### Requirement: Redact an API key written into a destination assistant's configuration +### Requirement: Redact a credential in an assistant difference -When `assistant compare`, `assistant verify` or `assistant import` prints either side of a difference, in the text or the JSON report, langstar SHALL print `` in place of the value of each key whose name ends in `api_key` when that value is a string. langstar SHALL match the name without regard to case. langstar SHALL redact the string both when the difference is at the key and when the difference is at an object that contains the key. +When `assistant compare`, `assistant verify` or `assistant import` prints either side of a difference, in text or JSON, langstar SHALL print `` in place of each string the credential check from "Refuse to export an assistant that holds a credential" flags. It SHALL redact the string when the difference is at the string or at an object that contains it, and whether the string came from the destination or from the file. #### Scenario: Destination configuration holds a literal API key @@ -240,9 +294,22 @@ When `assistant compare`, `assistant verify` or `assistant import` prints either - **AND** a user runs `langstar assistant compare support-bot.json --deployment my-agent-prod` - **THEN** langstar reports `/assistant/config/configurable/OPENAI_API_KEY` with `` in place of the string and exits with status 3 +#### Scenario: Destination context holds a connection string with a password + +- **WHEN** someone updated the destination assistant so `context.checkpoint_uri` holds `postgresql://app:@db.example.com/agent` +- **AND** a user runs `langstar assistant compare support-bot.json --deployment my-agent-prod -f json` +- **THEN** the text and the JSON report print `` at `/assistant/context/checkpoint_uri`, neither holds the password, and langstar exits with status 3 + +#### Scenario: Destination adds a configuration object that holds a password + +- **WHEN** the file's configuration has no `db` key +- **AND** someone updated the destination assistant so `config.configurable.db` holds `{"user": "app", "password": ""}` +- **AND** a user runs `langstar assistant compare support-bot.json --deployment my-agent-prod` +- **THEN** langstar reports `/assistant/config/configurable/db` with `user` unchanged and `` under `password`, and exits with status 3 + ### Requirement: Print a secret reference in a destination assistant's configuration unchanged -When `assistant compare`, `assistant verify` or `assistant import` prints either side of a difference, in the text or the JSON report, langstar SHALL print a LangChain secret reference unchanged when it is the value of a key whose name ends in `api_key`, so the report names the secret. A secret reference looks like `{"lc": 1, "type": "secret", "id": ["OPENAI_API_KEY"]}`, and langstar matches the key name without regard to case. +When `assistant compare`, `assistant verify` or `assistant import` prints either side of a difference, in the text or the JSON report, langstar SHALL print a LangChain secret reference unchanged under any key, so the report names the secret. A secret reference looks like `{"lc": 1, "type": "secret", "id": ["OPENAI_API_KEY"]}`. #### Scenario: Destination configuration reads its key from a secret diff --git a/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md index 165cb060d..b7384821f 100644 --- a/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md +++ b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md @@ -66,9 +66,9 @@ The export file SHALL list, under `requires.secrets`, the name of every secret t - **AND** a user runs `langstar prompt import my-prompt.json --workspace-id ` - **THEN** langstar creates no prompt, names `requires.secrets` and exits with status 1 -### Requirement: Refuse to export an API key written into a prompt's model +### Requirement: Refuse to export a prompt that holds a credential -When a chat model constructor in the manifest holds a string, rather than a secret reference, under a setting whose name ends in `api_key`, compared without regard to case, such as `openai_api_key` or `OPENAI_API_KEY`, `langstar prompt export` SHALL write no file and nothing to stdout, name that setting's JSON Pointer path without its value, and exit with status 1. +Before it writes, `langstar prompt export` SHALL run the credential check over every string in the `repo` and `commit` objects it would write, skipping LangChain secret references and empty strings. When the check flags a string, export SHALL write no file and nothing to stdout, list the JSON Pointer path of each flagged string and the rule that flagged it, without the string or any part of it, and exit with status 1. Export SHALL NOT redact a flagged string and write the rest of the file. #### Scenario: Model holds a literal API key @@ -76,6 +76,72 @@ When a chat model constructor in the manifest holds a string, rather than a secr - **AND** a user runs `langstar prompt export my-prompt --output my-prompt.json` - **THEN** langstar writes no file, names the path of `openai_api_key` without its value and exits with status 1 +#### Scenario: Model holds a literal API key under an upper-case name + +- **WHEN** the prompt's `ChatOpenAI` constructor holds a string under `OPENAI_API_KEY` +- **AND** a user runs `langstar prompt export my-prompt --output my-prompt.json` +- **THEN** langstar writes no file, names the path of `OPENAI_API_KEY` without its value and exits with status 1 + +#### Scenario: Readme holds a private key + +- **WHEN** the repo's readme holds a PEM block that begins `-----BEGIN PRIVATE KEY-----` +- **AND** a user runs `langstar prompt export my-prompt --output my-prompt.json` +- **THEN** langstar writes no file, names `/repo/readme` and exits with status 1 + +#### Scenario: Model reads its key from a secret + +- **WHEN** the prompt's chat model refers to the secret `OPENAI_API_KEY` under `openai_api_key` +- **AND** a user runs `langstar prompt export my-prompt --output my-prompt.json` +- **THEN** langstar writes the file with the secret reference and exits with status 0 + +### Requirement: Flag a prompt string by the name of its key + +The credential check SHALL flag a string when the key that holds it, or the array it is in, has a name whose words, split at `_`, `-`, `.`, spaces and lower-to-upper case changes and compared without regard to case, include `secret`, `token`, `password`, `passwd`, `pwd`, `credential`, `credentials`, `apikey`, `authorization` or `cookie`, or `key` right after `api`, `access`, `private`, `secret`, `signing` or `account`. + +#### Scenario: Model sends an authorization header + +- **WHEN** the prompt's chat model holds `default_headers` with `Authorization` set to `Bearer ` and a token +- **AND** a user runs `langstar prompt export my-prompt --output my-prompt.json` +- **THEN** langstar writes no file, names `/commit/manifest/kwargs/last/kwargs/default_headers/Authorization` without its value and exits with status 1 + +#### Scenario: Model sets a token limit + +- **WHEN** the prompt's chat model holds the number 1024 under `max_tokens` +- **AND** a user runs `langstar prompt export my-prompt --output my-prompt.json` +- **THEN** langstar writes the file with `max_tokens` unchanged and exits with status 0 + +### Requirement: Flag a prompt string by its shape under any key + +The credential check SHALL flag a string, under any key, that holds a bearer token, a `Basic ` authorization value, a PEM private key, a URL with a password in its user info, a connection string with a non-empty `Password=`, `Pwd=`, `AccountKey=`, `SharedAccessKey=` or `SharedAccessSignature=`, a JSON Web Token, or a token that starts with a provider prefix the design lists: `sk-`, `lsv2_`, `ghp_`, `gho_`, `ghs_`, `github_pat_`, `xoxb-`, `xoxp-`, `xoxa-`, `AKIA`, `ASIA`, `AIza` or `hf_`. + +#### Scenario: Model setting holds a token under a neutral name + +- **WHEN** the prompt's chat model holds a string that starts with `sk-` under `model_kwargs.extra` +- **AND** a user runs `langstar prompt export my-prompt` +- **THEN** stdout is empty, langstar names the path of `extra` with the rule that flagged it and exits with status 1 + +#### Scenario: Prompt metadata holds a connection string with a password + +- **WHEN** the manifest's `metadata` holds `postgresql://app:@db.example.com/prod` under `source_db` +- **AND** a user runs `langstar prompt export my-prompt --output my-prompt.json` +- **THEN** langstar writes no file, names the path of `source_db` as a URL with a password and exits with status 1 + +### Requirement: Let the user export a flagged string that holds no secret + +`langstar prompt export` SHALL take `--not-a-secret ` any number of times. For each pointer at a string the credential check flags, export SHALL write that string and print a warning to stderr naming the pointer. A flagged string at any path not named SHALL still make export refuse. When a pointer names a path the check does not flag, export SHALL write nothing, name the pointer and exit with status 1. No flag SHALL turn the check off. + +#### Scenario: Message template shows an example token + +- **WHEN** a message template holds `Authorization: Bearer ` followed by an example token +- **AND** a user runs `langstar prompt export my-prompt --output my-prompt.json --not-a-secret ` +- **THEN** langstar writes the file with the template unchanged, prints a warning naming the path to stderr and exits with status 0 + +#### Scenario: Exemption names a path the check does not flag + +- **WHEN** a user runs `langstar prompt export my-prompt --output my-prompt.json --not-a-secret /repo/description` +- **AND** the check flags nothing at `/repo/description` +- **THEN** langstar writes no file, names `/repo/description` and exits with status 1 + ### Requirement: Export, import and verify a prompt without a model When the manifest holds no chat model constructor, `langstar prompt export` SHALL write `requires.model_class` as null. Import and verify SHALL accept null when the manifest they read again holds no chat model, skip the saved model configuration check, report no note for it, and still run the secret check. @@ -249,9 +315,9 @@ After writing, `langstar prompt import` SHALL record the commit hash in the dest - **THEN** langstar lists that field as a difference and exits with status 3 - **AND** the destination record holds the new commit hash -### Requirement: Redact an API key written into a destination prompt's model +### Requirement: Redact a credential in a prompt difference -When `prompt compare`, `prompt verify` or `prompt import` prints either side of a difference, in the text or the JSON report, langstar SHALL print `` in place of the value of each key whose name ends in `api_key` when that value is a string. langstar SHALL match the name without regard to case. langstar SHALL redact the string both when the difference is at the key and when the difference is at an object that contains the key. +When `prompt compare`, `prompt verify` or `prompt import` prints either side of a difference, in text or JSON, langstar SHALL print `` in place of each string the credential check from "Refuse to export a prompt that holds a credential" flags. It SHALL redact the string when the difference is at the string or at an object that contains it, and whether the string came from the destination or from the file. #### Scenario: Destination model holds a literal API key @@ -266,9 +332,21 @@ When `prompt compare`, `prompt verify` or `prompt import` prints either side of - **AND** a user runs `langstar prompt compare my-prompt.json` - **THEN** langstar reports the model's path with `` in place of the string and exits with status 3 +#### Scenario: Destination model sends a bearer token in a header + +- **WHEN** someone saved the destination prompt with a model whose `default_headers.Authorization` holds `Bearer ` and a token +- **AND** a user runs `langstar prompt compare my-prompt.json -f json` +- **THEN** the text and the JSON report print `` at that header's path, neither holds the token, and langstar exits with status 3 + +#### Scenario: Destination model holds a token under a neutral name + +- **WHEN** someone saved the destination prompt with a model whose `model_kwargs.extra` holds a string that starts with `lsv2_` +- **AND** a user runs `langstar prompt compare my-prompt.json` +- **THEN** langstar reports that path with `` in place of the string and exits with status 3 + ### Requirement: Print a secret reference in a destination prompt's model unchanged -When `prompt compare`, `prompt verify` or `prompt import` prints either side of a difference, in the text or the JSON report, langstar SHALL print a LangChain secret reference unchanged when it is the value of a key whose name ends in `api_key`, so the report names the secret. A secret reference looks like `{"lc": 1, "type": "secret", "id": ["OPENAI_API_KEY"]}`, and langstar matches the key name without regard to case. +When `prompt compare`, `prompt verify` or `prompt import` prints either side of a difference, in the text or the JSON report, langstar SHALL print a LangChain secret reference unchanged under any key, so the report names the secret. A secret reference looks like `{"lc": 1, "type": "secret", "id": ["OPENAI_API_KEY"]}`. #### Scenario: Destination adds a model that reads its key from a secret diff --git a/openspec/changes/export-import-prompts-and-assistants/tasks.md b/openspec/changes/export-import-prompts-and-assistants/tasks.md index 6ed514222..4d994952c 100644 --- a/openspec/changes/export-import-prompts-and-assistants/tasks.md +++ b/openspec/changes/export-import-prompts-and-assistants/tasks.md @@ -40,20 +40,31 @@ Each integration test in this group carries `#[cfg_attr(not(feature = "integrati - Unit tests on nested objects, arrays, missing keys, and keys holding `/` or `~`, which the report escapes as RFC 6901 requires, so `a/b` appears as `/a~1b` and `a~b` as `/a~0b`. - A test that a changed `id` array inside `commit.manifest`, such as `ChatOpenAI` replaced by `ChatAnthropic`, is reported at its path. - A test that a destination whose `commit.model_config` and `commit.model_provider` differ from the file's, while the manifest matches, is reported as a match. -- [ ] 4.3 Add the shared result report, which the compare, verify and import commands for both prompts and assistants print through, and the exit statuses 0, 3 and 1. The report prints JSON to stdout and progress to stderr. Before it prints either side of a difference, the report puts `` in place of the value of each key whose name ends in `api_key`, matched without regard to case, when that value is a string, and prints a secret reference under such a key unchanged. It does this both when the difference is at the key and when it is at an object that contains the key. Leave clap's status 2 for usage errors. Verify with unit tests of the report, of the exit status each result gives, of a difference at `openai_api_key` where the destination holds a string, of a difference at `/commit/manifest/kwargs/last` where the file has no model and the destination's model holds a string under `openai_api_key`, of a difference at `/assistant/config/configurable/openai_api_key` where the destination holds a string, and of a difference at `/assistant/config/configurable/OPENAI_API_KEY` where the destination holds a string. Each redaction test asserts that the text and the `-f json` report print `` and that neither holds the string. Add a test of a difference at `/commit/manifest/kwargs/last` where the destination's model refers to the secret `OPENAI_API_KEY`, asserting that the text and the `-f json` report both print the secret reference with `OPENAI_API_KEY` in it, and that neither prints ``. Add a test of a difference at `/assistant/config/configurable/openai_api_key` where the destination refers to the secret `OPENAI_API_KEY`, asserting that the text and the `-f json` report both print the secret reference with `OPENAI_API_KEY` in it, and that neither prints ``. -- [ ] 4.4 Add the secret reference walk that lists secret names from a manifest and returns the JSON Pointer path of each chat model setting whose name ends in `api_key`, compared without regard to case, and holds a string instead of a secret reference, which `prompt export` from 5.1 refuses with status 1; verify with unit tests on a copy of the `prompt_with_model` manifest from the LangSmith Python SDK's integration tests, which `references/langsmith-sdk.md` links, on a manifest with no secret reference, and on manifests whose `openai_api_key` and whose `OPENAI_API_KEY` hold a string, asserting that the walk returns that setting's path and that the error message holds the path and not the string +- [ ] 4.3 Add the shared result report, which the compare, verify and import commands for both prompts and assistants print through, and the exit statuses 0, 3 and 1. The report prints JSON to stdout and progress to stderr. Before it prints either side of a difference, the report puts `` in place of every string the credential check from 4.4 flags, by either rule, and prints a secret reference unchanged under any key. It does this both when the difference is at the string and when it is at an object that contains it, for strings from the destination and from the file. Leave clap's status 2 for usage errors. Verify with these tests: + - Unit tests of the report and of the exit status each result gives. + - Redaction tests of a difference at `openai_api_key` where the destination holds a string; at `/commit/manifest/kwargs/last` where the file has no model and the destination's model holds a string under `openai_api_key`; at `/assistant/config/configurable/openai_api_key` and at `/assistant/config/configurable/OPENAI_API_KEY` where the destination holds a string; at a model's `default_headers/Authorization` holding `Bearer ` and a token; at a model's `model_kwargs/extra` holding a string that starts with `lsv2_`; at `/assistant/context/checkpoint_uri` holding a `postgresql://` URL with a password; and at `/assistant/config/configurable/db` where the destination adds `{"user": "app", "password": ""}`. Each asserts that the text and the `-f json` report print `` at the flagged string, that neither holds the string or any part of it, and, for the `db` object, that `user` prints unchanged. + - Tests of a difference at `/commit/manifest/kwargs/last` where the destination's model refers to the secret `OPENAI_API_KEY`, and at `/assistant/config/configurable/openai_api_key` where the destination refers to that secret, asserting that the text and the `-f json` report both print the secret reference with `OPENAI_API_KEY` in it, and that neither prints ``. +- [ ] 4.4 Add the secret reference walk that lists secret names from a manifest, and the credential check that export, compare, verify and import share. The check walks every string under a JSON value, skips secret references and empty strings, and returns the JSON Pointer path of each flagged string with the rule that flagged it, the key-name rule or one of the value shapes in design.md, never the string. Keep the word list and the value shapes in one module with a doc comment that names each. Verify with these tests: + - Unit tests of the secret reference walk on a copy of the `prompt_with_model` manifest from the LangSmith Python SDK's integration tests, which `references/langsmith-sdk.md` links, and on a manifest with no secret reference. + - Key-name tests that flag a string under `openai_api_key`, `OPENAI_API_KEY`, `x-api-key`, `Authorization`, `Cookie`, `access_token`, `refresh_token`, `client_secret`, `db_password`, `privateKey`, `signingKey` and `credentials`, and under an array whose key is `api_keys`, and that do not flag a string under `max_tokens`, `cache_key`, `author` or `keywords`, or a number or boolean under a flagged name. + - Value-shape tests under a neutral key such as `note` that flag `Bearer ` followed by a 20-character token, `Basic ` followed by 20 base64 characters, a PEM block for each of `PRIVATE KEY`, `RSA PRIVATE KEY` and `OPENSSH PRIVATE KEY`, `postgresql://app:@host/db`, `redis://:@host:6379`, `Server=db;User Id=app;Password=`, an Azure string with `AccountKey=`, a JSON Web Token, and a token with each prefix the design lists. Each asserts the rule named in the result. + - Tests that do not flag ordinary text: `Basic instructions follow.`, `Bearer in mind`, `https://example.com/path`, `postgresql://host/db` with no password, `sk-` on its own, and `-----BEGIN PUBLIC KEY-----`. + - Tests that skip a secret reference under `openai_api_key` and an empty string under `password`, and that escape a flagged key holding `/` or `~` in the path as RFC 6901 requires. + - A test that the error message for each flagged string holds the path and the rule, and does not hold the string or any eight-character run of it. ## 5. prompt export -- [ ] 5.1 Add `langstar prompt export` with `--commit`, `--output` and the scoping flags. Verify with these tests: +- [ ] 5.1 Add `langstar prompt export` with `--commit`, `--output`, `--not-a-secret` and the scoping flags, and run the credential check from 4.4 over the `repo` and `commit` objects before writing. Verify with these tests: - Integration tests that create four prompts through the SDK (a `json_schema` prompt with no model, a `function_calling` prompt with a nested schema, a prompt with several messages and a placeholder, and a prompt with a model and metadata), export each through the CLI, and assert every field of the file against the SDK's read of the commit. The file for the prompt with a model lists `OPENAI_API_KEY` under `requires.secrets` and `ChatOpenAI` under `requires.model_class`, and holds no secret value. The file for the `json_schema` prompt holds an empty `requires.secrets` and null under `requires.model_class`. - An integration test that exports, with `--commit prod`, a prompt whose `prod` tag points at an earlier commit, and asserts that `source.commit_hash` and the manifest are that earlier commit's. - An error test that exports a handle the workspace has no prompt for, and asserts that it exits with status 1, names the handle and writes no file. - A CLI test that runs `prompt export` without `--output`, parses all of stdout as one JSON document, and asserts that its `source.commit_hash` matches the SDK's read and that any progress message is on stderr. - An integration test that gives one commit two tags, exports it twice, and asserts that the two files are identical. - - A unit test in `cli/src/commands/prompt.rs` that serves, through a `mockito` server and `LangchainClient::with_base_urls`, a commit whose `ChatOpenAI` constructor holds a string under `openai_api_key`, and another under `OPENAI_API_KEY`, calls the export code on each with `--output my-prompt.json`, and asserts that it returns an error that exits with status 1 and names the setting's path without the string, and that it writes no file. + - Unit tests in `cli/src/commands/prompt.rs` that serve, through a `mockito` server and `LangchainClient::with_base_urls`, one commit for each export refusal scenario in `specs/prompt/spec.md`: a `ChatOpenAI` constructor with a string under `openai_api_key`, and another under `OPENAI_API_KEY`; `default_headers.Authorization` holding `Bearer ` and a token; `model_kwargs.extra` holding a string that starts with `sk-`; manifest `metadata.source_db` holding a `postgresql://` URL with a password; and a repo readme holding a PEM private key. Each calls the export code with `--output my-prompt.json`, and once without `--output`, and asserts that it exits with status 1, names the path and the rule without the string, writes no file and prints nothing to stdout. + - A unit test that serves a commit whose model refers to the secret `OPENAI_API_KEY` under `openai_api_key`, and asserts that export writes the file with the secret reference. + - Unit tests of `--not-a-secret`: a message template holding `Authorization: Bearer ` and an example token, exported with `--not-a-secret` at the template's path, writes the file with the template unchanged and prints a warning naming the path to stderr; the same commit with a second flagged string at another path still exits with status 1 and names only that path; and `--not-a-secret /repo/description`, where nothing is flagged, exits with status 1, names the pointer and writes no file. - A unit test in `cli/src/commands/prompt.rs` that serves a prompt through a `mockito` server and `LangchainClient::with_base_urls`, calls the export code with no workspace ID set, and asserts that `source.workspace_id` holds the repo's `tenant_id`. -- [ ] 5.2 Document `prompt export` in `docs/usage/` and the README. The page says that export refuses a prompt whose model holds an API key as a literal string, names that setting and writes no file, and that a credential kept under any other name, such as a header, still reaches the file; tells users to keep credentials in the workspace's secrets; and tells them to read an export file before they commit it. Verify that each documented command runs as written. +- [ ] 5.2 Document `prompt export` in `docs/usage/` and the README. The page lists the key names and value shapes the credential check flags, says that export then names each path and writes no file, shows `--not-a-secret` for a flagged string that holds no secret, and says that a credential with none of those names or shapes, such as a random string under a key called `x`, still reaches the file. It tells users to keep credentials in the workspace's secrets, and to read an export file before they commit it. Verify that each documented command runs as written. ## 6. prompt compare and verify @@ -101,15 +112,17 @@ Each integration test in this group carries `#[cfg_attr(not(feature = "integrati ## 8. assistant export -- [ ] 8.1 Add `langstar assistant export` with `--output`, `--prompt-ref` and the refusal of a string under a key whose name ends in `api_key`, compared without regard to case, anywhere in the configuration, context or metadata. Verify with these tests: +- [ ] 8.1 Add `langstar assistant export` with `--output`, `--prompt-ref` and `--not-a-secret`, and run the credential check from 4.4 over the `assistant` object before writing. Verify with these tests: - An integration test that creates an assistant with configuration, context and metadata through the SDK and asserts every field of the exported file. - An integration test that exports, with `--prompt-ref /config/configurable/system_prompt`, an assistant whose value there is `support-system:prod`, and asserts that the file holds that pointer and `support-system:prod` under `requires.prompts`. - An integration test that exports the same assistant version twice and asserts that the two files are identical. - An error test for a pointer that names nothing. - An error test that exporting an assistant ID the deployment lacks exits with status 1, names the ID and writes no file. - - A unit test in `cli/src/commands/assistant.rs` that serves, through a `mockito` server and `LangchainClient::with_base_urls`, three assistants: one whose `config.configurable.OPENAI_API_KEY` holds a string, one whose `context.anthropic_api_key` holds a string, and one whose `metadata.openai_api_key` holds a string. It calls the export code on each with `--output support-bot.json` and asserts that each call returns an error that exits with status 1 and names the key's JSON Pointer path without the string, and that it writes no file. + - Unit tests in `cli/src/commands/assistant.rs` that serve, through a `mockito` server and `LangchainClient::with_base_urls`, one assistant for each export refusal scenario in `specs/assistant/spec.md`: `config.configurable.OPENAI_API_KEY`, `context.anthropic_api_key` and `metadata.openai_api_key` holding a string; `context.github_access_token` holding a string; `config.configurable.db.password` holding a string; `config.configurable.checkpoint_uri` holding a `postgresql://` URL with a password; `config.configurable.tool_headers.Authorization` holding `Bearer ` and a token; and `metadata.deploy_note` holding an OpenSSH private key. Each calls the export code with `--output support-bot.json`, and once without `--output`, and asserts that it exits with status 1, names the JSON Pointer path and the rule without the string, writes no file and prints nothing to stdout. + - A unit test that serves an assistant whose `config.configurable.openai_api_key` holds a secret reference, and asserts that export writes the file with the reference. + - A unit test that exports an assistant whose `config.configurable.token_budget_policy` holds `strict` with `--not-a-secret /config/configurable/token_budget_policy`, and asserts that the file holds `strict` there and stderr names the path, and one without the flag that asserts status 1. - A CLI test that runs `assistant export` without `--output`, parses all of stdout as one JSON document, and asserts that its `source.assistant_id` is the exported assistant's ID and that any progress message is on stderr. -- [ ] 8.2 Document assistant export. The page says that export refuses an assistant whose configuration, context or metadata holds a string under a key whose name ends in `api_key`, names that key's path and writes no file, and that a credential kept under any other name still reaches the file; tells users to keep credentials in the deployment's environment variables or the workspace's secrets; and tells them to read an export file before they commit it. Verify that each documented command runs as written. +- [ ] 8.2 Document assistant export. The page links the prompt export page's list of what the credential check flags, says that export then names each path and writes no file, shows `--not-a-secret`, and says that a credential with none of those names or shapes still reaches the file. It tells users to keep credentials in the deployment's environment variables or the workspace's secrets, and to read an export file before they commit it. Verify that each documented command runs as written. ## 9. assistant compare and verify From 67fa07dd4272e70c84ae4384a81ffa479b79a5a5 Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Tue, 27 Oct 2026 16:24:20 -0400 Subject: [PATCH 68/90] =?UTF-8?q?=F0=9F=93=9A=20docs(openspec):=20ship=20t?= =?UTF-8?q?he=20SDK=20field=20additions=20as=20feat=20commits,=20as=20the?= =?UTF-8?q?=20release=20docs=20and=20#726=20did,=20and=20drop=20open=20que?= =?UTF-8?q?stion=209?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit docs/dev/procedures.md counts changed signatures and removed fields as breaking, not added ones; docs/dev/git-scm-conventions.md and docs/dev/ci-cd.md give feat a minor bump; the Prepare Release workflow picks the version from commit types; release.toml keeps langstar-sdk off crates.io. Co-Authored-By: Claude Opus 5.5 --- .../design.md | 21 ++++++++++++------- .../proposal.md | 2 +- .../tasks.md | 2 +- 3 files changed, 15 insertions(+), 10 deletions(-) diff --git a/openspec/changes/export-import-prompts-and-assistants/design.md b/openspec/changes/export-import-prompts-and-assistants/design.md index ad6cb7268..7617e7d92 100644 --- a/openspec/changes/export-import-prompts-and-assistants/design.md +++ b/openspec/changes/export-import-prompts-and-assistants/design.md @@ -128,11 +128,23 @@ The LangSmith API has no documented way to run a prompt with its model, so a sho [#728 Implement CRUD command structure (create, get, update) for prompt commands](https://github.com/codekiln/langstar/issues/728) plans `prompt create`, `get` and `update` commands that edit a prompt's text, model or structured output one field at a time. Export and import read and write the whole prompt at once, through the same `PromptClient` calls in `sdk/src/prompts.rs` that those commands will use, so a field the SDK reads for export is there for `prompt get` too. +### SDK field additions ship as `✨ feat` commits + +The SDK tasks in group 3 add public fields to `Prompt`, `CommitManifestResponse`, `CommitData`, `CommitRequest`, `Assistant`, `CreateAssistantRequest`, `UpdateAssistantRequest` and `AssistantSearchRequest`, and add new calls. Their commits are `✨ feat(sdk)` commits with no `BREAKING CHANGE:` footer and no `!` before the colon, and no pull request in this change edits the version in `Cargo.toml`. That follows the repository's release policy: + +- `docs/dev/git-scm-conventions.md:26` gives `✨ feat` a minor version bump, and `docs/dev/git-scm-conventions.md:28` keeps the major bump for `🚨 BREAKING CHANGE`. The table at `docs/dev/ci-cd.md:209` adds a `BREAKING CHANGE:` footer and `!` before the colon to the major bump. +- `docs/dev/procedures.md:116` names changing function signatures and removing fields as breaking SDK changes. Adding a field or a call is neither. +- The Prepare Release workflow, `.github/workflows/prepare-release.yml:62`, picks the bump by running the bump-release skill's `analyze_commits.py` over the commits since the last tag, and writes the new version itself. git-cliff writes the changelog from the same commit messages, so nobody writes release notes or a version by hand. +- `release.toml:5` sets `publish = false`: langstar ships GitHub releases and does not publish `langstar-sdk` to crates.io, so the only crate known to build these structs is the CLI in this repository. +- The same kind of change has shipped this way before. [4a57ccb ✨ feat(assistant): add description field support from Agent Server API (#726)](https://github.com/codekiln/langstar/commit/4a57ccb6802e988a1210ba8a8b1434a9bf9a803e) added `description` to `Assistant`, `CreateAssistantRequest` and `UpdateAssistantRequest` with no footer, and v2.2.0 released it as a minor version after v2.1.2. + +The structs stay without `#[non_exhaustive]`. That attribute would stop the CLI and the SDK's integration tests from building `Assistant`, `CommitRequest`, `CreateAssistantRequest` and `UpdateAssistantRequest` with struct literals, which is a change `docs/dev/procedures.md:116` would count as breaking. + ## Risks / Trade-offs - [LangSmith may store a pushed model differently from the source] → Import reads the prompt back and reports any difference, and in [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8) a developer checks the round trip against the live API before anyone builds the commands. - [Pushing a manifest identical to the latest commit may return 409] → Import compares first and pushes nothing when the prompt matches. -- [The SDK changes for export and import add public fields to `Prompt`, `CommitManifestResponse`, `CommitData`, `CommitRequest`, `Assistant` and the assistant request structs, so a crate that builds one of those structs with a struct literal or destructures it with a pattern that names every field stops compiling] → Each pull request that adds a field updates every struct literal of that struct in this repository. [Should the SDK field additions count as a breaking change?](#9---should-the-sdk-field-additions-count-as-a-breaking-change) asks codekiln whether those pull requests also carry a `BREAKING CHANGE:` footer, which `docs/dev/git-scm-conventions.md` says makes the release a major version. +- [The SDK changes for export and import add public fields to `Prompt`, `CommitManifestResponse`, `CommitData`, `CommitRequest`, `Assistant` and the assistant request structs, so a crate that builds one of those structs with a struct literal stops compiling] → Langstar does not publish `langstar-sdk` to crates.io, and each pull request that adds a field updates every struct literal of that struct in this repository, as `docs/dev/procedures.md:116` asks. [SDK field additions ship as `✨ feat` commits](#sdk-field-additions-ship-as--feat-commits) gives the release policy. - [A prompt or an assistant may hold a credential, such as an API key, a token, a password, an authorization header, a connection string with a password or a private key, which export would copy into a file meant for Git] → Export runs the credential check from [Export refuses to write a file that holds a credential](#export-refuses-to-write-a-file-that-holds-a-credential) and, when it flags a string, writes nothing, names each path and exits with status 1. Only `--not-a-secret ` lets a flagged string through, one path at a time. - [The credential check misses a credential with no telling key name or value shape, such as a random string under a key called `x`] → The export docs in [export-import-prompts-and-assistants Task 5.2 Document prompt export](tasks.md#L67) and [export-import-prompts-and-assistants Task 8.2 Document assistant export](tasks.md#L125) list what the check catches and what it cannot, tell users to keep credentials in the workspace's secrets or the deployment's environment variables, and tell them to read an export file before committing it. - [The check flags a string that holds no secret, such as an example token in a message template] → Export names the path and the rule, and the user passes `--not-a-secret` with that path after reading the string. @@ -201,10 +213,3 @@ None. > [#807 Export, import, compare, and verify prompts and assistants across LangSmith environments](https://github.com/codekiln/langstar/issues/807) asks import to check the destination's dependencies: the secrets a prompt's model reads, and an assistant's graph and referenced prompts. The draft runs those checks before import writes and writes nothing when one fails, so a user never has to find and delete a copy that cannot run. The cost is order: the destination workspace needs its secrets before a prompt import, and its prompts before an assistant import. The other choice writes the copy anyway and reports the failed checks with status 3 afterwards, which lets a user import in any order and fix the destination later. Recommendation: stop before writing. - -### 9 - Should the SDK field additions count as a breaking change? - -> Context from the drafting agent, for question 9. -> A new public field breaks only a crate that builds the struct with a struct literal or destructures it with a pattern that names every field; nothing in this repository destructures these structs that way. The only crate known to use the SDK is langstar's own CLI, and each pull request that adds a field updates the CLI's struct literals. `sdk/Cargo.toml` leaves the crate publishable, though, so an outside crate could break, and `docs/dev/git-scm-conventions.md` gives every API break a `BREAKING CHANGE:` footer, which makes the release a major version. The structs that gain fields are `Prompt` (which `PromptClient::get` returns), `CommitManifestResponse` (which `PromptClient::get_commit` returns), `CommitData`, `CommitRequest`, `Assistant` and the assistant request structs. Marking them `#[non_exhaustive]` would stop every crate except the SDK itself from building them with struct literals, even with `..Default::default()`. langstar's own CLI builds `Assistant`, `CommitRequest`, `CreateAssistantRequest` and `UpdateAssistantRequest` with struct literals today, and the SDK's integration tests build all of them except `Assistant` the same way, so the SDK would need a constructor for each of those four structs before the CLI and those tests compile again. After that one break, a new field would break nothing. [🚨 BREAKING CHANGE(sdk): move annotation-queue run calls to the platform items API · Pull Request #815](https://github.com/codekiln/langstar/pull/815) removes SDK functions and changes return types, marks itself breaking, and leaves that choice to codekiln too, so one answer to this question can set the policy for both pull requests. Recommendation: treat the additions as `feat` changes with no `BREAKING CHANGE:` footer, and name the new fields in the release notes. - - diff --git a/openspec/changes/export-import-prompts-and-assistants/proposal.md b/openspec/changes/export-import-prompts-and-assistants/proposal.md index 2c43c17e4..85cab1e6a 100644 --- a/openspec/changes/export-import-prompts-and-assistants/proposal.md +++ b/openspec/changes/export-import-prompts-and-assistants/proposal.md @@ -31,7 +31,7 @@ A prompt belongs to a workspace and an assistant belongs to a deployment, but bo ## Impact -- SDK: `sdk/src/prompts.rs` gains the prompt calls export and import need: a commit read that asks for the model and returns it on `CommitManifestResponse`, a commit create that sends a description and returns the new commit's `id`, a commit list, calls that list, create, move and delete commit tags, and a repo update. `Prompt` gains the repo's readme, tags and `tenant_id`, and export fills `source.workspace_id` from `tenant_id`. `sdk/src/assistants.rs` gains the `context` and `version` fields, `assistant_id` and `if_exists` on create, `graph_id` on update, and a search by graph ID. Each new public field breaks a crate outside this repository that builds that SDK struct with a struct literal or destructures it with a pattern that names every field; codekiln decides in [Should the SDK field additions count as a breaking change?](design.md#9---should-the-sdk-field-additions-count-as-a-breaking-change) whether that makes the release a major version. +- SDK: `sdk/src/prompts.rs` gains the prompt calls export and import need: a commit read that asks for the model and returns it on `CommitManifestResponse`, a commit create that sends a description and returns the new commit's `id`, a commit list, calls that list, create, move and delete commit tags, and a repo update. `Prompt` gains the repo's readme, tags and `tenant_id`, and export fills `source.workspace_id` from `tenant_id`. `sdk/src/assistants.rs` gains the `context` and `version` fields, `assistant_id` and `if_exists` on create, `graph_id` on update, and a search by graph ID. These are `✨ feat(sdk)` commits with no `BREAKING CHANGE:` footer, so the Prepare Release workflow makes the next release a minor version, as [SDK field additions ship as `✨ feat` commits](design.md#sdk-field-additions-ship-as--feat-commits) explains from `docs/dev/git-scm-conventions.md`, `docs/dev/ci-cd.md` and `docs/dev/procedures.md`. - CLI: new subcommands in `cli/src/commands/prompt.rs` and `cli/src/commands/assistant.rs`, and a shared module for the export file, the destination record, comparison and the exit status. - Tests need a second workspace, a second deployment and one model provider key for the structured output check. codekiln creates the workspace and the key, and langstar's tests read them through `fnox.toml` and the CI workflow; the destination deployment fixture from [export-import-prompts-and-assistants Task 2.4 Add a destination deployment fixture](tasks.md#L19) creates the deployment whenever the cleanup workflow has deleted it, as the source fixture does today. Nobody knows yet whether the control plane accepts a second deployment from this repository. In [export-import-prompts-and-assistants Task 2.3 Register the test graph under a second graph ID](tasks.md#L18), a developer deploys the new `langgraph.json` once to find out, before anyone builds the fixture. - Docs: a new usage page on moving prompts and assistants between environments. diff --git a/openspec/changes/export-import-prompts-and-assistants/tasks.md b/openspec/changes/export-import-prompts-and-assistants/tasks.md index 4d994952c..efb4e971c 100644 --- a/openspec/changes/export-import-prompts-and-assistants/tasks.md +++ b/openspec/changes/export-import-prompts-and-assistants/tasks.md @@ -20,7 +20,7 @@ Each numbered group below is one pull request, merged in the order listed, excep ## 3. SDK reads and writes the fields export and import need -Each task in this group adds public fields to SDK structs and updates every struct literal of those structs in this repository, so the workspace still compiles. The structs are `Prompt`, `CommitManifestResponse`, `CommitData`, `CommitRequest`, `Assistant`, `CreateAssistantRequest`, `UpdateAssistantRequest` and `AssistantSearchRequest`. A crate outside this repository that builds one of them with a struct literal stops compiling too, and codekiln decides in [Should the SDK field additions count as a breaking change?](design.md#9---should-the-sdk-field-additions-count-as-a-breaking-change) whether this group's pull request carries a `BREAKING CHANGE:` footer. +Each task in this group adds public fields to SDK structs and updates every struct literal of those structs in this repository, so the workspace still compiles. The structs are `Prompt`, `CommitManifestResponse`, `CommitData`, `CommitRequest`, `Assistant`, `CreateAssistantRequest`, `UpdateAssistantRequest` and `AssistantSearchRequest`. Commit these tasks as `✨ feat(sdk)` with no `BREAKING CHANGE:` footer and leave the version in `Cargo.toml` alone, as [SDK field additions ship as `✨ feat` commits](design.md#sdk-field-additions-ship-as--feat-commits) explains; the Prepare Release workflow picks the version. Each integration test in this group carries `#[cfg_attr(not(feature = "integration-tests"), ignore)]`, as the tests in `sdk/tests/structured_prompts_integration_test.rs` do, so that the SDK Integration Tests job in `.github/workflows/ci.yml` and `mise run check` both run it. From beed3c5e2fc1b6dccb2278d163eeed62545a624e Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Tue, 27 Oct 2026 16:39:47 -0400 Subject: [PATCH 69/90] =?UTF-8?q?=F0=9F=93=9A=20docs(openspec):=20say=20pl?= =?UTF-8?q?ainly=20what=20the=20credential=20check=20flags=20and=20when=20?= =?UTF-8?q?export=20refuses?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Lead the design paragraph with what export does, split the "never redacts and writes the rest" sentence that read two ways, state the --not-a-secret rules without stacked negatives, and rename the requirements and scenarios whose subjects could not act. Co-Authored-By: Claude Opus 5.5 --- .../design.md | 18 +++++++++--------- .../proposal.md | 2 +- .../specs/assistant/spec.md | 12 ++++++------ .../specs/prompt/spec.md | 10 +++++----- .../tasks.md | 10 +++++----- 5 files changed, 26 insertions(+), 26 deletions(-) diff --git a/openspec/changes/export-import-prompts-and-assistants/design.md b/openspec/changes/export-import-prompts-and-assistants/design.md index 7617e7d92..58d4fc6c5 100644 --- a/openspec/changes/export-import-prompts-and-assistants/design.md +++ b/openspec/changes/export-import-prompts-and-assistants/design.md @@ -57,18 +57,18 @@ Import, compare and verify read a file only when its `format_version` is 1 and i ### Export refuses to write a file that holds a credential -An export file is meant for Git, and a secret pushed to Git stays in its history after anyone deletes it. So before it writes anything, export runs a credential check over every string it would write in the `repo`, `commit` and `assistant` objects. For a prompt that covers the manifest with each model setting, header, `model_kwargs` entry, message template and `metadata` value in it, `commit.model_config`, the commit description and the repo's description and readme. For an assistant it covers the name, description, configuration, context and metadata. The check skips a LangChain secret reference, `{"lc": 1, "type": "secret", "id": ["OPENAI_API_KEY"]}`, because it holds only a secret's name, and skips the empty string. +Before it writes anything, export runs a credential check over every string it would put in the `repo`, `commit` and `assistant` objects, because a secret pushed to Git stays in the repository's history after someone deletes the file. For a prompt, the check reads the manifest with each model setting, header, `model_kwargs` entry, message template and `metadata` value in it, `commit.model_config`, the commit description and the repo's description and readme. For an assistant it covers the name, description, configuration, context and metadata. The check skips a LangChain secret reference, `{"lc": 1, "type": "secret", "id": ["OPENAI_API_KEY"]}`, because it holds only a secret's name, and skips the empty string. -The check flags a string by two rules. +The check flags a string when the name of its key or the shape of its value marks it as a credential. -- **The key's name.** Langstar splits the name of the key that holds the string, or that holds the array the string is in, into words at `_`, `-`, `.`, spaces and each change from a lower-case to an upper-case letter, and compares the words without regard to case. A name with one of the words `secret`, `token`, `password`, `passwd`, `pwd`, `credential`, `credentials`, `apikey`, `authorization` or `cookie`, or with the word `key` right after `api`, `access`, `private`, `secret`, `signing` or `account`, is flagged. So `openai_api_key`, `OPENAI_API_KEY`, `x-api-key`, `Authorization`, `access_token`, `client_secret`, `db_password` and `privateKey` all are, and `max_tokens` and `cache_key` are not. -- **The value's shape**, under any key. Langstar flags a string that holds a bearer token (`Bearer ` followed by at least 16 token characters) or a `Basic ` value followed by at least 16 base64 characters; a PEM private key (`-----BEGIN` and `PRIVATE KEY-----`); a URL with a password in it, `scheme://user:password@host` or `scheme://:password@host`, which covers database and queue connection strings; a key-value connection string with a non-empty `Password=`, `Pwd=`, `AccountKey=`, `SharedAccessKey=` or `SharedAccessSignature=`; a JSON Web Token, three base64url parts that start with `eyJ`; or a token with a provider's published prefix, which are `sk-` (OpenAI and Anthropic), `lsv2_` (LangSmith), `ghp_`, `gho_`, `ghs_` and `github_pat_` (GitHub), `xoxb-`, `xoxp-` and `xoxa-` (Slack), `AKIA` and `ASIA` followed by 16 upper-case letters or digits (AWS), `AIza` (Google) and `hf_` (Hugging Face), each followed by enough characters to rule out an ordinary word. +- **By key name:** the check flags a string whose key, or the key of the array it sits in, holds one of the words `secret`, `token`, `password`, `passwd`, `pwd`, `credential`, `credentials`, `apikey`, `authorization` or `cookie`, or holds `key` right after `api`, `access`, `private`, `secret`, `signing` or `account`. It splits a key name into words at `_`, `-`, `.`, spaces and each change from a lower-case to an upper-case letter, and ignores case. It flags `openai_api_key`, `OPENAI_API_KEY`, `x-api-key`, `Authorization`, `access_token`, `client_secret`, `db_password` and `privateKey`, and passes `max_tokens` and `cache_key`. +- **By value shape**, under any key: the check flags a string that holds a bearer token (`Bearer ` followed by at least 16 token characters) or a `Basic ` value followed by at least 16 base64 characters; a PEM private key (`-----BEGIN` and `PRIVATE KEY-----`); a URL with a password in it, `scheme://user:password@host` or `scheme://:password@host`, which covers database and queue connection strings; a key-value connection string with a non-empty `Password=`, `Pwd=`, `AccountKey=`, `SharedAccessKey=` or `SharedAccessSignature=`; a JSON Web Token, three base64url parts that start with `eyJ`; or a token with a provider's published prefix, which are `sk-` (OpenAI and Anthropic), `lsv2_` (LangSmith), `ghp_`, `gho_`, `ghs_` and `github_pat_` (GitHub), `xoxb-`, `xoxp-` and `xoxa-` (Slack), `AKIA` and `ASIA` followed by 16 upper-case letters or digits (AWS), `AIza` (Google) and `hf_` (Hugging Face), each followed by enough characters to rule out an ordinary word. -When the check flags any string, export writes no file and nothing to stdout, lists the JSON Pointer path of each flagged string with the rule that flagged it, such as `key name` or `URL with a password`, never the string or any part of it, and exits with status 1. Export never redacts a credential and writes the rest. A redacted file would import a prompt or assistant that cannot authenticate, and a reviewer reading the file in a pull request would not learn that the source holds a credential. The refusal asks the user to move the credential into the workspace's secrets, which a prompt's model reads through a secret reference, or into the deployment's environment variables, which a graph reads. +When the check flags any string, export writes no file and nothing to stdout, lists the JSON Pointer path of each flagged string with the rule that flagged it, such as `key name` or `URL with a password`, leaves the string and every part of it out of the message, and exits with status 1. Export refuses instead of writing `` in place of the credential, because a redacted file would import a prompt or assistant that cannot authenticate, and would hide from a reviewer reading the file in a pull request that the source holds a credential. The refusal asks the user to move the credential into the workspace's secrets, which a prompt's model reads through a secret reference, or into the deployment's environment variables, which a graph reads. -A string the check flags may hold no secret, such as a message template that shows `Bearer ` followed by an example token. `--not-a-secret `, which a user can give more than once, tells export to write the string at that one path, and export prints a warning to stderr naming each such path. A pointer at a string the check does not flag makes export write nothing, name the pointer and exit with status 1, so a script cannot keep a stale exemption. No flag turns the check off. +Some flagged strings are harmless, such as a message template that shows `Bearer ` followed by an example token. A user who has read such a string passes `--not-a-secret `, once for each path, and export writes the string at that path and prints a warning to stderr naming the path. When a pointer names a string the check passes, export refuses as well: it names the pointer and exits with status 1, so a script that still passes the pointer after the string changed stops at the export. Every export runs the check. -The check cannot recognize a credential that has none of these names or shapes, such as a random string under a key called `x`. The export docs tell users to keep credentials in secrets and environment variables, and to read an export file before committing it. +A credential under an ordinary key name with an ordinary-looking value, such as a random string under a key called `x`, passes the check and reaches the file. The export docs tell users to keep credentials in secrets and environment variables, and to read an export file before committing it. ### Export asks LangSmith for the model @@ -96,7 +96,7 @@ Without `--update`, import lists the differences and exits with status 3. `--dry ### Compare reports differences by JSON path -Compare reads the prompt or assistant from the destination, puts it in the shape of the export file's `repo`, `commit` and `assistant` objects, and lists every value that differs from the file. Compare checks every key in those objects, including the chat model's settings and the `id` arrays inside a manifest that name the prompt and model classes, such as `ChatOpenAI`. It skips `commit.model_config` and `commit.model_provider`, because LangSmith fills both in itself when import pushes a commit, and a destination workspace can fill them in differently from the source. Export leaves the values the server sets, such as IDs, version numbers, timestamps, owner and like counts, out of those objects, so they never show up as differences. The source environment's workspace, deployment and IDs sit in `source`, so they never show up as differences against a copy. Each difference names its JSON Pointer path and both values. Before the report prints either side of a difference, it replaces with `` every string that export's credential check flags, by either rule, whether the difference is at that string or at an object that contains it. A credential someone pasted into the destination prompt or the destination assistant, or into the file, therefore stays out of the terminal and the CI log. The report prints a LangChain secret reference, such as `{"lc": 1, "type": "secret", "id": ["OPENAI_API_KEY"]}`, unchanged, because the reference holds only the secret's name, never its value. The report redacts rather than refuses because it writes to a terminal or log, not to a file meant for Git, and the user still needs to see where the difference is. For a prompt, the comparison covers the commit description, the commit tags and the commit each one points at, any tag on the compared commit that the file lacks, and the repo's description, readme, tags and visibility as well as the manifest. Import runs the same comparison after it writes, so a field LangSmith or the Agent Server drops shows up as a difference. +Compare reads the prompt or assistant from the destination, puts it in the shape of the export file's `repo`, `commit` and `assistant` objects, and lists every value that differs from the file. Compare checks every key in those objects, including the chat model's settings and the `id` arrays inside a manifest that name the prompt and model classes, such as `ChatOpenAI`. It skips `commit.model_config` and `commit.model_provider`, because LangSmith fills both in itself when import pushes a commit, and a destination workspace can fill them in differently from the source. Export leaves the values the server sets, such as IDs, version numbers, timestamps, owner and like counts, out of those objects, so they never show up as differences. The source environment's workspace, deployment and IDs sit in `source`, so they never show up as differences against a copy. Each difference names its JSON Pointer path and both values. Before the report prints either side of a difference, it replaces with `` every string that export's credential check flags, by either rule, whether the difference is at that string or at an object that contains it. A credential someone pasted into the destination prompt or the destination assistant, or into the file, therefore stays out of the terminal and the CI log. The report prints a LangChain secret reference, such as `{"lc": 1, "type": "secret", "id": ["OPENAI_API_KEY"]}`, unchanged, because the reference holds only the secret's name, never its value. The report prints to a terminal or a CI log, where the user needs to see where each difference is, so it redacts the string and goes on reporting. For a prompt, the comparison covers the commit description, the commit tags and the commit each one points at, any tag on the compared commit that the file lacks, and the repo's description, readme, tags and visibility as well as the manifest. Import runs the same comparison after it writes, so a field LangSmith or the Agent Server drops shows up as a difference. ### Verify checks what the copy needs to run @@ -146,7 +146,7 @@ The structs stay without `#[non_exhaustive]`. That attribute would stop the CLI - [Pushing a manifest identical to the latest commit may return 409] → Import compares first and pushes nothing when the prompt matches. - [The SDK changes for export and import add public fields to `Prompt`, `CommitManifestResponse`, `CommitData`, `CommitRequest`, `Assistant` and the assistant request structs, so a crate that builds one of those structs with a struct literal stops compiling] → Langstar does not publish `langstar-sdk` to crates.io, and each pull request that adds a field updates every struct literal of that struct in this repository, as `docs/dev/procedures.md:116` asks. [SDK field additions ship as `✨ feat` commits](#sdk-field-additions-ship-as--feat-commits) gives the release policy. - [A prompt or an assistant may hold a credential, such as an API key, a token, a password, an authorization header, a connection string with a password or a private key, which export would copy into a file meant for Git] → Export runs the credential check from [Export refuses to write a file that holds a credential](#export-refuses-to-write-a-file-that-holds-a-credential) and, when it flags a string, writes nothing, names each path and exits with status 1. Only `--not-a-secret ` lets a flagged string through, one path at a time. -- [The credential check misses a credential with no telling key name or value shape, such as a random string under a key called `x`] → The export docs in [export-import-prompts-and-assistants Task 5.2 Document prompt export](tasks.md#L67) and [export-import-prompts-and-assistants Task 8.2 Document assistant export](tasks.md#L125) list what the check catches and what it cannot, tell users to keep credentials in the workspace's secrets or the deployment's environment variables, and tell them to read an export file before committing it. +- [A credential under an ordinary key name with an ordinary-looking value, such as a random string under a key called `x`, passes the credential check] → The export docs in [export-import-prompts-and-assistants Task 5.2 Document prompt export](tasks.md#L67) and [export-import-prompts-and-assistants Task 8.2 Document assistant export](tasks.md#L125) list what the check catches and what it cannot, tell users to keep credentials in the workspace's secrets or the deployment's environment variables, and tell them to read an export file before committing it. - [The check flags a string that holds no secret, such as an example token in a message template] → Export names the path and the rule, and the user passes `--not-a-secret` with that path after reading the string. - [A tag in the destination may already point at another commit] → `--update` moves the tag, and `--dry-run` lists the move. - [`--update` deletes a tag on the matching destination commit that the file lacks, and a graph on a deployment may read that tag] → `--dry-run` lists each delete, and `assistant verify` reports a prompt tag an assistant refers to that the workspace no longer has. diff --git a/openspec/changes/export-import-prompts-and-assistants/proposal.md b/openspec/changes/export-import-prompts-and-assistants/proposal.md index 85cab1e6a..f79f0b2a0 100644 --- a/openspec/changes/export-import-prompts-and-assistants/proposal.md +++ b/openspec/changes/export-import-prompts-and-assistants/proposal.md @@ -14,7 +14,7 @@ A prompt belongs to a workspace and an assistant belongs to a deployment, but bo - `langstar prompt import --workspace-id ` creates the prompt in the destination workspace, or adds a commit to an existing prompt, updates its repo details and creates, moves or deletes its commit tags when the user passes `--update`. An import that stopped before it pushed the first commit finishes on the next run, and one that stopped before it created the commit tags finishes with `--update`. `--dry-run` shows what it would do. After writing, it reads the prompt back and reports any difference. - `langstar prompt compare ` reports the differences between the file and a prompt in a workspace. `langstar prompt verify ` also checks that the workspace has the secrets the prompt's model reads, and notes when the workspace has no saved model configuration for the model's class, such as `ChatOpenAI`. - `langstar assistant export`, `import`, `compare` and `verify` do the same for assistants on a deployment. The file holds the name, description, graph ID, configuration, context, metadata and any prompt names the assistant refers to. `verify` checks that the destination deployment has the graph and that the destination workspace has the referenced prompts. -- Both exports refuse to write a file that holds a credential. Before writing, export checks every string it would write for a key name such as `api_key`, `token`, `password`, `secret` or `Authorization`, and for a value shaped like a bearer token, a private key, a URL or connection string with a password, a JSON Web Token or a provider's API key. When one matches, export names each path, writes nothing and exits with status 1. `--not-a-secret ` lets one flagged string through after the user has read it. Compare, verify and import print `` in place of the same strings. +- `prompt export` and `assistant export` write no file when the prompt or assistant holds a credential. Before writing, export checks every string for a key name such as `api_key`, `token`, `password`, `secret` or `Authorization`, and for a value shaped like a bearer token, a private key, a URL or connection string with a password, a JSON Web Token or a provider's API key. Export names the path of each flagged string and exits with status 1. A user who has read a flagged string and found it harmless passes `--not-a-secret ` to write it. Compare, verify and import print `` in place of the same strings. - Import records each destination ID (the new commit hash, or the new assistant ID) in a separate file next to the export. When a user imports the same file again, langstar reads that file to find the assistant it created earlier instead of creating another one, and to tell whether someone pushed a commit to the prompt since the last import. - Import runs the dependency checks of `verify` before it writes, and writes nothing when one fails. - `compare`, `verify` and `import` share one exit status scheme, 0 for a match, 3 for a difference or a failed check and 1 for any other error, and print a JSON report with `-f json`. Status 2 stays clap's usage error. diff --git a/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md b/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md index 60ac01f6d..dfb088c5f 100644 --- a/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md +++ b/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md @@ -23,7 +23,7 @@ ### Requirement: Refuse to export an assistant that holds a credential -Before it writes, `langstar assistant export` SHALL run the credential check over every string at any depth of the `assistant` object, skipping secret references and empty strings. When the check flags a string, export SHALL write no file and nothing to stdout, list the JSON Pointer path and rule of each flagged string, without the string or any part of it, and exit with status 1. Export SHALL NOT redact a flagged string and write the rest. +Before it writes, `langstar assistant export` SHALL run the credential check over every string at any depth of the `assistant` object, skipping secret references and empty strings. When the check flags a string, export SHALL write no file and nothing to stdout, list the JSON Pointer path and rule of each flagged string, without the string or any part of it, and exit with status 1. Export SHALL NOT write a file with `` in place of a flagged string. #### Scenario: Configuration holds a literal API key @@ -75,11 +75,11 @@ The credential check SHALL flag a string, under any key, that holds a bearer tok - **AND** a user runs `langstar assistant export --deployment my-agent` - **THEN** stdout is empty, langstar names `/metadata/deploy_note` as a private key and exits with status 1 -### Requirement: Let the user export a flagged assistant string that holds no secret +### Requirement: Write each flagged assistant string the user names with --not-a-secret -`langstar assistant export` SHALL take `--not-a-secret ` any number of times. For each pointer at a string the credential check flags, export SHALL write that string and print a warning to stderr naming the pointer. A flagged string at any path not named SHALL still make export refuse. When a pointer names a path the check does not flag, export SHALL write nothing, name the pointer and exit with status 1. No flag SHALL turn the check off. +`langstar assistant export` SHALL take `--not-a-secret ` any number of times. For each pointer at a string the credential check flags, export SHALL write that string and print a warning to stderr naming the pointer. A flagged string at any path not named SHALL still make export refuse. When a pointer names a string the check passes, export SHALL write nothing, name the pointer and exit with status 1. Export SHALL run the check on every run. -#### Scenario: Configuration names a token budget +#### Scenario: User writes a token budget setting with --not-a-secret - **WHEN** the assistant's `config.configurable.token_budget_policy` holds the string `strict` - **AND** a user runs `langstar assistant export --deployment my-agent --output support-bot.json --not-a-secret /config/configurable/token_budget_policy` @@ -278,7 +278,7 @@ After writing, `langstar assistant import` SHALL record the assistant ID in the - **WHEN** a user runs `langstar -f json assistant verify support-bot.json --deployment my-agent-prod` - **THEN** stdout holds one JSON object with `status`, `differences` and `checks` -### Requirement: Redact a credential in an assistant difference +### Requirement: Redact a credential when reporting how an assistant differs When `assistant compare`, `assistant verify` or `assistant import` prints either side of a difference, in text or JSON, langstar SHALL print `` in place of each string the credential check from "Refuse to export an assistant that holds a credential" flags. It SHALL redact the string when the difference is at the string or at an object that contains it, and whether the string came from the destination or from the file. @@ -300,7 +300,7 @@ When `assistant compare`, `assistant verify` or `assistant import` prints either - **AND** a user runs `langstar assistant compare support-bot.json --deployment my-agent-prod -f json` - **THEN** the text and the JSON report print `` at `/assistant/context/checkpoint_uri`, neither holds the password, and langstar exits with status 3 -#### Scenario: Destination adds a configuration object that holds a password +#### Scenario: Destination configuration has a new object that holds a password - **WHEN** the file's configuration has no `db` key - **AND** someone updated the destination assistant so `config.configurable.db` holds `{"user": "app", "password": ""}` diff --git a/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md index b7384821f..a27dbca77 100644 --- a/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md +++ b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md @@ -68,7 +68,7 @@ The export file SHALL list, under `requires.secrets`, the name of every secret t ### Requirement: Refuse to export a prompt that holds a credential -Before it writes, `langstar prompt export` SHALL run the credential check over every string in the `repo` and `commit` objects it would write, skipping LangChain secret references and empty strings. When the check flags a string, export SHALL write no file and nothing to stdout, list the JSON Pointer path of each flagged string and the rule that flagged it, without the string or any part of it, and exit with status 1. Export SHALL NOT redact a flagged string and write the rest of the file. +Before it writes, `langstar prompt export` SHALL run the credential check over every string in the `repo` and `commit` objects it would write, skipping LangChain secret references and empty strings. When the check flags a string, export SHALL write no file and nothing to stdout, list the JSON Pointer path of each flagged string and the rule that flagged it, without the string or any part of it, and exit with status 1. Export SHALL NOT write a file with `` in place of a flagged string. #### Scenario: Model holds a literal API key @@ -126,9 +126,9 @@ The credential check SHALL flag a string, under any key, that holds a bearer tok - **AND** a user runs `langstar prompt export my-prompt --output my-prompt.json` - **THEN** langstar writes no file, names the path of `source_db` as a URL with a password and exits with status 1 -### Requirement: Let the user export a flagged string that holds no secret +### Requirement: Write each flagged prompt string the user names with --not-a-secret -`langstar prompt export` SHALL take `--not-a-secret ` any number of times. For each pointer at a string the credential check flags, export SHALL write that string and print a warning to stderr naming the pointer. A flagged string at any path not named SHALL still make export refuse. When a pointer names a path the check does not flag, export SHALL write nothing, name the pointer and exit with status 1. No flag SHALL turn the check off. +`langstar prompt export` SHALL take `--not-a-secret ` any number of times. For each pointer at a string the credential check flags, export SHALL write that string and print a warning to stderr naming the pointer. A flagged string at any path not named SHALL still make export refuse. When a pointer names a string the check passes, export SHALL write nothing, name the pointer and exit with status 1. Export SHALL run the check on every run. #### Scenario: Message template shows an example token @@ -136,7 +136,7 @@ The credential check SHALL flag a string, under any key, that holds a bearer tok - **AND** a user runs `langstar prompt export my-prompt --output my-prompt.json --not-a-secret ` - **THEN** langstar writes the file with the template unchanged, prints a warning naming the path to stderr and exits with status 0 -#### Scenario: Exemption names a path the check does not flag +#### Scenario: User passes --not-a-secret for a string the check passes - **WHEN** a user runs `langstar prompt export my-prompt --output my-prompt.json --not-a-secret /repo/description` - **AND** the check flags nothing at `/repo/description` @@ -315,7 +315,7 @@ After writing, `langstar prompt import` SHALL record the commit hash in the dest - **THEN** langstar lists that field as a difference and exits with status 3 - **AND** the destination record holds the new commit hash -### Requirement: Redact a credential in a prompt difference +### Requirement: Redact a credential when reporting how a prompt differs When `prompt compare`, `prompt verify` or `prompt import` prints either side of a difference, in text or JSON, langstar SHALL print `` in place of each string the credential check from "Refuse to export a prompt that holds a credential" flags. It SHALL redact the string when the difference is at the string or at an object that contains it, and whether the string came from the destination or from the file. diff --git a/openspec/changes/export-import-prompts-and-assistants/tasks.md b/openspec/changes/export-import-prompts-and-assistants/tasks.md index efb4e971c..d49316ce3 100644 --- a/openspec/changes/export-import-prompts-and-assistants/tasks.md +++ b/openspec/changes/export-import-prompts-and-assistants/tasks.md @@ -44,11 +44,11 @@ Each integration test in this group carries `#[cfg_attr(not(feature = "integrati - Unit tests of the report and of the exit status each result gives. - Redaction tests of a difference at `openai_api_key` where the destination holds a string; at `/commit/manifest/kwargs/last` where the file has no model and the destination's model holds a string under `openai_api_key`; at `/assistant/config/configurable/openai_api_key` and at `/assistant/config/configurable/OPENAI_API_KEY` where the destination holds a string; at a model's `default_headers/Authorization` holding `Bearer ` and a token; at a model's `model_kwargs/extra` holding a string that starts with `lsv2_`; at `/assistant/context/checkpoint_uri` holding a `postgresql://` URL with a password; and at `/assistant/config/configurable/db` where the destination adds `{"user": "app", "password": ""}`. Each asserts that the text and the `-f json` report print `` at the flagged string, that neither holds the string or any part of it, and, for the `db` object, that `user` prints unchanged. - Tests of a difference at `/commit/manifest/kwargs/last` where the destination's model refers to the secret `OPENAI_API_KEY`, and at `/assistant/config/configurable/openai_api_key` where the destination refers to that secret, asserting that the text and the `-f json` report both print the secret reference with `OPENAI_API_KEY` in it, and that neither prints ``. -- [ ] 4.4 Add the secret reference walk that lists secret names from a manifest, and the credential check that export, compare, verify and import share. The check walks every string under a JSON value, skips secret references and empty strings, and returns the JSON Pointer path of each flagged string with the rule that flagged it, the key-name rule or one of the value shapes in design.md, never the string. Keep the word list and the value shapes in one module with a doc comment that names each. Verify with these tests: +- [ ] 4.4 Add the secret reference walk that lists secret names from a manifest, and the credential check that export, compare, verify and import share. The check walks every string under a JSON value, skips secret references and empty strings, and returns the JSON Pointer path of each flagged string with the rule that flagged it, the key-name rule or one of the value shapes in [Export refuses to write a file that holds a credential](design.md#export-refuses-to-write-a-file-that-holds-a-credential), and leaves the string out. Keep the word list and the value shapes in one module with a doc comment that names each. Verify with these tests: - Unit tests of the secret reference walk on a copy of the `prompt_with_model` manifest from the LangSmith Python SDK's integration tests, which `references/langsmith-sdk.md` links, and on a manifest with no secret reference. - - Key-name tests that flag a string under `openai_api_key`, `OPENAI_API_KEY`, `x-api-key`, `Authorization`, `Cookie`, `access_token`, `refresh_token`, `client_secret`, `db_password`, `privateKey`, `signingKey` and `credentials`, and under an array whose key is `api_keys`, and that do not flag a string under `max_tokens`, `cache_key`, `author` or `keywords`, or a number or boolean under a flagged name. + - Key-name tests that flag a string under `openai_api_key`, `OPENAI_API_KEY`, `x-api-key`, `Authorization`, `Cookie`, `access_token`, `refresh_token`, `client_secret`, `db_password`, `privateKey`, `signingKey` and `credentials`, and under an array whose key is `api_keys`, and that pass a string under `max_tokens`, `cache_key`, `author` or `keywords`, and a number or boolean under a flagged name. - Value-shape tests under a neutral key such as `note` that flag `Bearer ` followed by a 20-character token, `Basic ` followed by 20 base64 characters, a PEM block for each of `PRIVATE KEY`, `RSA PRIVATE KEY` and `OPENSSH PRIVATE KEY`, `postgresql://app:@host/db`, `redis://:@host:6379`, `Server=db;User Id=app;Password=`, an Azure string with `AccountKey=`, a JSON Web Token, and a token with each prefix the design lists. Each asserts the rule named in the result. - - Tests that do not flag ordinary text: `Basic instructions follow.`, `Bearer in mind`, `https://example.com/path`, `postgresql://host/db` with no password, `sk-` on its own, and `-----BEGIN PUBLIC KEY-----`. + - Tests that pass ordinary text: `Basic instructions follow.`, `Bearer in mind`, `https://example.com/path`, `postgresql://host/db` with no password, `sk-` on its own, and `-----BEGIN PUBLIC KEY-----`. - Tests that skip a secret reference under `openai_api_key` and an empty string under `password`, and that escape a flagged key holding `/` or `~` in the path as RFC 6901 requires. - A test that the error message for each flagged string holds the path and the rule, and does not hold the string or any eight-character run of it. @@ -64,7 +64,7 @@ Each integration test in this group carries `#[cfg_attr(not(feature = "integrati - A unit test that serves a commit whose model refers to the secret `OPENAI_API_KEY` under `openai_api_key`, and asserts that export writes the file with the secret reference. - Unit tests of `--not-a-secret`: a message template holding `Authorization: Bearer ` and an example token, exported with `--not-a-secret` at the template's path, writes the file with the template unchanged and prints a warning naming the path to stderr; the same commit with a second flagged string at another path still exits with status 1 and names only that path; and `--not-a-secret /repo/description`, where nothing is flagged, exits with status 1, names the pointer and writes no file. - A unit test in `cli/src/commands/prompt.rs` that serves a prompt through a `mockito` server and `LangchainClient::with_base_urls`, calls the export code with no workspace ID set, and asserts that `source.workspace_id` holds the repo's `tenant_id`. -- [ ] 5.2 Document `prompt export` in `docs/usage/` and the README. The page lists the key names and value shapes the credential check flags, says that export then names each path and writes no file, shows `--not-a-secret` for a flagged string that holds no secret, and says that a credential with none of those names or shapes, such as a random string under a key called `x`, still reaches the file. It tells users to keep credentials in the workspace's secrets, and to read an export file before they commit it. Verify that each documented command runs as written. +- [ ] 5.2 Document `prompt export` in `docs/usage/` and the README. The page lists the key names and value shapes the credential check flags, says that export then names each path and writes no file, shows `--not-a-secret` for a flagged string that holds no secret, and says that a credential under an ordinary key name with an ordinary-looking value, such as a random string under a key called `x`, passes the check and reaches the file. It tells users to keep credentials in the workspace's secrets, and to read an export file before they commit it. Verify that each documented command runs as written. ## 6. prompt compare and verify @@ -122,7 +122,7 @@ Each integration test in this group carries `#[cfg_attr(not(feature = "integrati - A unit test that serves an assistant whose `config.configurable.openai_api_key` holds a secret reference, and asserts that export writes the file with the reference. - A unit test that exports an assistant whose `config.configurable.token_budget_policy` holds `strict` with `--not-a-secret /config/configurable/token_budget_policy`, and asserts that the file holds `strict` there and stderr names the path, and one without the flag that asserts status 1. - A CLI test that runs `assistant export` without `--output`, parses all of stdout as one JSON document, and asserts that its `source.assistant_id` is the exported assistant's ID and that any progress message is on stderr. -- [ ] 8.2 Document assistant export. The page links the prompt export page's list of what the credential check flags, says that export then names each path and writes no file, shows `--not-a-secret`, and says that a credential with none of those names or shapes still reaches the file. It tells users to keep credentials in the deployment's environment variables or the workspace's secrets, and to read an export file before they commit it. Verify that each documented command runs as written. +- [ ] 8.2 Document assistant export. The page links the prompt export page's list of what the credential check flags, says that export then names each path and writes no file, shows `--not-a-secret`, and says that a credential under an ordinary key name with an ordinary-looking value passes the check and reaches the file. It tells users to keep credentials in the deployment's environment variables or the workspace's secrets, and to read an export file before they commit it. Verify that each documented command runs as written. ## 9. assistant compare and verify From fc01799a35d08189651e10ea0911d014289d07fb Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Tue, 27 Oct 2026 16:40:30 -0400 Subject: [PATCH 70/90] =?UTF-8?q?=F0=9F=93=9A=20docs(openspec):=20name=20w?= =?UTF-8?q?ho=20commits=20the=20SDK=20fields=20and=20link=20the=20release?= =?UTF-8?q?=20docs=20the=20design=20cites?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Retitle the release decision so it says who acts and what release follows, replace the group 3 ordinal with a linked task pointer, drop "ship", and turn each cited release-doc line into a link pinned to main. Co-Authored-By: Claude Opus 5.5 --- .../design.md | 18 +++++++++--------- .../proposal.md | 2 +- .../tasks.md | 2 +- 3 files changed, 11 insertions(+), 11 deletions(-) diff --git a/openspec/changes/export-import-prompts-and-assistants/design.md b/openspec/changes/export-import-prompts-and-assistants/design.md index 58d4fc6c5..0843d2c98 100644 --- a/openspec/changes/export-import-prompts-and-assistants/design.md +++ b/openspec/changes/export-import-prompts-and-assistants/design.md @@ -128,23 +128,23 @@ The LangSmith API has no documented way to run a prompt with its model, so a sho [#728 Implement CRUD command structure (create, get, update) for prompt commands](https://github.com/codekiln/langstar/issues/728) plans `prompt create`, `get` and `update` commands that edit a prompt's text, model or structured output one field at a time. Export and import read and write the whole prompt at once, through the same `PromptClient` calls in `sdk/src/prompts.rs` that those commands will use, so a field the SDK reads for export is there for `prompt get` too. -### SDK field additions ship as `✨ feat` commits +### Developers commit the new SDK fields as `✨ feat(sdk)`, so the next release is a minor version -The SDK tasks in group 3 add public fields to `Prompt`, `CommitManifestResponse`, `CommitData`, `CommitRequest`, `Assistant`, `CreateAssistantRequest`, `UpdateAssistantRequest` and `AssistantSearchRequest`, and add new calls. Their commits are `✨ feat(sdk)` commits with no `BREAKING CHANGE:` footer and no `!` before the colon, and no pull request in this change edits the version in `Cargo.toml`. That follows the repository's release policy: +In [export-import-prompts-and-assistants Task group 3 SDK reads and writes the fields export and import need](tasks.md#L21), developers add public fields to `Prompt`, `CommitManifestResponse`, `CommitData`, `CommitRequest`, `Assistant`, `CreateAssistantRequest`, `UpdateAssistantRequest` and `AssistantSearchRequest`, and add new calls. They commit that work as `✨ feat(sdk)` with no `BREAKING CHANGE:` footer and no `!` after the type, and leave the version in `Cargo.toml` to the Prepare Release workflow. The repository's release docs ask for this: -- `docs/dev/git-scm-conventions.md:26` gives `✨ feat` a minor version bump, and `docs/dev/git-scm-conventions.md:28` keeps the major bump for `🚨 BREAKING CHANGE`. The table at `docs/dev/ci-cd.md:209` adds a `BREAKING CHANGE:` footer and `!` before the colon to the major bump. -- `docs/dev/procedures.md:116` names changing function signatures and removing fields as breaking SDK changes. Adding a field or a call is neither. -- The Prepare Release workflow, `.github/workflows/prepare-release.yml:62`, picks the bump by running the bump-release skill's `analyze_commits.py` over the commits since the last tag, and writes the new version itself. git-cliff writes the changelog from the same commit messages, so nobody writes release notes or a version by hand. -- `release.toml:5` sets `publish = false`: langstar ships GitHub releases and does not publish `langstar-sdk` to crates.io, so the only crate known to build these structs is the CLI in this repository. -- The same kind of change has shipped this way before. [4a57ccb ✨ feat(assistant): add description field support from Agent Server API (#726)](https://github.com/codekiln/langstar/commit/4a57ccb6802e988a1210ba8a8b1434a9bf9a803e) added `description` to `Assistant`, `CreateAssistantRequest` and `UpdateAssistantRequest` with no footer, and v2.2.0 released it as a minor version after v2.1.2. +- [docs/dev/git-scm-conventions.md:26](https://github.com/codekiln/langstar/blob/ab158b14160cf7803a79384f88756911bcd21857/docs/dev/git-scm-conventions.md#L26) gives `✨ feat` a minor version bump, and [docs/dev/git-scm-conventions.md:28](https://github.com/codekiln/langstar/blob/ab158b14160cf7803a79384f88756911bcd21857/docs/dev/git-scm-conventions.md#L28) keeps the major bump for `🚨 BREAKING CHANGE`. The table at [docs/dev/ci-cd.md:209](https://github.com/codekiln/langstar/blob/ab158b14160cf7803a79384f88756911bcd21857/docs/dev/ci-cd.md#L209) adds a `BREAKING CHANGE:` footer and `!` before the colon to the major bump. +- [docs/dev/procedures.md:116](https://github.com/codekiln/langstar/blob/ab158b14160cf7803a79384f88756911bcd21857/docs/dev/procedures.md#L116) names changing function signatures and removing fields as breaking SDK changes. Adding a field or a call is neither. +- The Prepare Release workflow, [.github/workflows/prepare-release.yml:62](https://github.com/codekiln/langstar/blob/ab158b14160cf7803a79384f88756911bcd21857/.github/workflows/prepare-release.yml#L62), picks the bump by running the bump-release skill's `analyze_commits.py` over the commits since the last tag, and writes the new version itself. git-cliff writes the changelog from the same commit messages, so nobody writes release notes or a version by hand. +- [release.toml:5](https://github.com/codekiln/langstar/blob/ab158b14160cf7803a79384f88756911bcd21857/release.toml#L5) sets `publish = false`, which keeps `langstar-sdk` off crates.io, so the only crate known to build these structs is the CLI in this repository. +- [4a57ccb ✨ feat(assistant): add description field support from Agent Server API (#726)](https://github.com/codekiln/langstar/commit/4a57ccb6802e988a1210ba8a8b1434a9bf9a803e) added `description` to `Assistant`, `CreateAssistantRequest` and `UpdateAssistantRequest` with no footer, and langstar released it in v2.2.0, a minor version after v2.1.2. -The structs stay without `#[non_exhaustive]`. That attribute would stop the CLI and the SDK's integration tests from building `Assistant`, `CommitRequest`, `CreateAssistantRequest` and `UpdateAssistantRequest` with struct literals, which is a change `docs/dev/procedures.md:116` would count as breaking. +Developers leave the structs as they are, without `#[non_exhaustive]`, because adding that attribute would stop the CLI and the SDK's integration tests from building `Assistant`, `CommitRequest`, `CreateAssistantRequest` and `UpdateAssistantRequest` with struct literals, which [docs/dev/procedures.md:116](https://github.com/codekiln/langstar/blob/ab158b14160cf7803a79384f88756911bcd21857/docs/dev/procedures.md#L116) counts as a breaking change. ## Risks / Trade-offs - [LangSmith may store a pushed model differently from the source] → Import reads the prompt back and reports any difference, and in [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8) a developer checks the round trip against the live API before anyone builds the commands. - [Pushing a manifest identical to the latest commit may return 409] → Import compares first and pushes nothing when the prompt matches. -- [The SDK changes for export and import add public fields to `Prompt`, `CommitManifestResponse`, `CommitData`, `CommitRequest`, `Assistant` and the assistant request structs, so a crate that builds one of those structs with a struct literal stops compiling] → Langstar does not publish `langstar-sdk` to crates.io, and each pull request that adds a field updates every struct literal of that struct in this repository, as `docs/dev/procedures.md:116` asks. [SDK field additions ship as `✨ feat` commits](#sdk-field-additions-ship-as--feat-commits) gives the release policy. +- [The SDK changes for export and import add public fields to `Prompt`, `CommitManifestResponse`, `CommitData`, `CommitRequest`, `Assistant` and the assistant request structs, so a crate that builds one of those structs with a struct literal stops compiling] → `release.toml` keeps `langstar-sdk` off crates.io, and the developer who adds a field updates every struct literal of that struct in the same pull request, as [docs/dev/procedures.md:116](https://github.com/codekiln/langstar/blob/ab158b14160cf7803a79384f88756911bcd21857/docs/dev/procedures.md#L116) asks. [Developers commit the new SDK fields as `✨ feat(sdk)`, so the next release is a minor version](#developers-commit-the-new-sdk-fields-as--featsdk-so-the-next-release-is-a-minor-version) gives the release policy. - [A prompt or an assistant may hold a credential, such as an API key, a token, a password, an authorization header, a connection string with a password or a private key, which export would copy into a file meant for Git] → Export runs the credential check from [Export refuses to write a file that holds a credential](#export-refuses-to-write-a-file-that-holds-a-credential) and, when it flags a string, writes nothing, names each path and exits with status 1. Only `--not-a-secret ` lets a flagged string through, one path at a time. - [A credential under an ordinary key name with an ordinary-looking value, such as a random string under a key called `x`, passes the credential check] → The export docs in [export-import-prompts-and-assistants Task 5.2 Document prompt export](tasks.md#L67) and [export-import-prompts-and-assistants Task 8.2 Document assistant export](tasks.md#L125) list what the check catches and what it cannot, tell users to keep credentials in the workspace's secrets or the deployment's environment variables, and tell them to read an export file before committing it. - [The check flags a string that holds no secret, such as an example token in a message template] → Export names the path and the rule, and the user passes `--not-a-secret` with that path after reading the string. diff --git a/openspec/changes/export-import-prompts-and-assistants/proposal.md b/openspec/changes/export-import-prompts-and-assistants/proposal.md index f79f0b2a0..cfb8b4a18 100644 --- a/openspec/changes/export-import-prompts-and-assistants/proposal.md +++ b/openspec/changes/export-import-prompts-and-assistants/proposal.md @@ -31,7 +31,7 @@ A prompt belongs to a workspace and an assistant belongs to a deployment, but bo ## Impact -- SDK: `sdk/src/prompts.rs` gains the prompt calls export and import need: a commit read that asks for the model and returns it on `CommitManifestResponse`, a commit create that sends a description and returns the new commit's `id`, a commit list, calls that list, create, move and delete commit tags, and a repo update. `Prompt` gains the repo's readme, tags and `tenant_id`, and export fills `source.workspace_id` from `tenant_id`. `sdk/src/assistants.rs` gains the `context` and `version` fields, `assistant_id` and `if_exists` on create, `graph_id` on update, and a search by graph ID. These are `✨ feat(sdk)` commits with no `BREAKING CHANGE:` footer, so the Prepare Release workflow makes the next release a minor version, as [SDK field additions ship as `✨ feat` commits](design.md#sdk-field-additions-ship-as--feat-commits) explains from `docs/dev/git-scm-conventions.md`, `docs/dev/ci-cd.md` and `docs/dev/procedures.md`. +- SDK: `sdk/src/prompts.rs` gains the prompt calls export and import need: a commit read that asks for the model and returns it on `CommitManifestResponse`, a commit create that sends a description and returns the new commit's `id`, a commit list, calls that list, create, move and delete commit tags, and a repo update. `Prompt` gains the repo's readme, tags and `tenant_id`, and export fills `source.workspace_id` from `tenant_id`. `sdk/src/assistants.rs` gains the `context` and `version` fields, `assistant_id` and `if_exists` on create, `graph_id` on update, and a search by graph ID. Developers commit these fields as `✨ feat(sdk)` with no `BREAKING CHANGE:` footer, so the Prepare Release workflow makes the next release a minor version; [Developers commit the new SDK fields as `✨ feat(sdk)`, so the next release is a minor version](design.md#developers-commit-the-new-sdk-fields-as--featsdk-so-the-next-release-is-a-minor-version) cites the release docs. - CLI: new subcommands in `cli/src/commands/prompt.rs` and `cli/src/commands/assistant.rs`, and a shared module for the export file, the destination record, comparison and the exit status. - Tests need a second workspace, a second deployment and one model provider key for the structured output check. codekiln creates the workspace and the key, and langstar's tests read them through `fnox.toml` and the CI workflow; the destination deployment fixture from [export-import-prompts-and-assistants Task 2.4 Add a destination deployment fixture](tasks.md#L19) creates the deployment whenever the cleanup workflow has deleted it, as the source fixture does today. Nobody knows yet whether the control plane accepts a second deployment from this repository. In [export-import-prompts-and-assistants Task 2.3 Register the test graph under a second graph ID](tasks.md#L18), a developer deploys the new `langgraph.json` once to find out, before anyone builds the fixture. - Docs: a new usage page on moving prompts and assistants between environments. diff --git a/openspec/changes/export-import-prompts-and-assistants/tasks.md b/openspec/changes/export-import-prompts-and-assistants/tasks.md index d49316ce3..7d430b772 100644 --- a/openspec/changes/export-import-prompts-and-assistants/tasks.md +++ b/openspec/changes/export-import-prompts-and-assistants/tasks.md @@ -20,7 +20,7 @@ Each numbered group below is one pull request, merged in the order listed, excep ## 3. SDK reads and writes the fields export and import need -Each task in this group adds public fields to SDK structs and updates every struct literal of those structs in this repository, so the workspace still compiles. The structs are `Prompt`, `CommitManifestResponse`, `CommitData`, `CommitRequest`, `Assistant`, `CreateAssistantRequest`, `UpdateAssistantRequest` and `AssistantSearchRequest`. Commit these tasks as `✨ feat(sdk)` with no `BREAKING CHANGE:` footer and leave the version in `Cargo.toml` alone, as [SDK field additions ship as `✨ feat` commits](design.md#sdk-field-additions-ship-as--feat-commits) explains; the Prepare Release workflow picks the version. +Each task in this group adds public fields to SDK structs and updates every struct literal of those structs in this repository, so the workspace still compiles. The structs are `Prompt`, `CommitManifestResponse`, `CommitData`, `CommitRequest`, `Assistant`, `CreateAssistantRequest`, `UpdateAssistantRequest` and `AssistantSearchRequest`. Commit these tasks as `✨ feat(sdk)` with no `BREAKING CHANGE:` footer and leave the version in `Cargo.toml` alone, as [Developers commit the new SDK fields as `✨ feat(sdk)`, so the next release is a minor version](design.md#developers-commit-the-new-sdk-fields-as--featsdk-so-the-next-release-is-a-minor-version) explains; the Prepare Release workflow picks the version. Each integration test in this group carries `#[cfg_attr(not(feature = "integration-tests"), ignore)]`, as the tests in `sdk/tests/structured_prompts_integration_test.rs` do, so that the SDK Integration Tests job in `.github/workflows/ci.yml` and `mise run check` both run it. From 1b6b6915b37fa519be038360861e350712d96f0a Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Tue, 27 Oct 2026 16:40:47 -0400 Subject: [PATCH 71/90] =?UTF-8?q?=F0=9F=93=9A=20docs(openspec):=20open=20t?= =?UTF-8?q?he=20SDK=20release=20decision=20with=20what=20developers=20comm?= =?UTF-8?q?it?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 5.5 --- openspec/changes/export-import-prompts-and-assistants/design.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/openspec/changes/export-import-prompts-and-assistants/design.md b/openspec/changes/export-import-prompts-and-assistants/design.md index 0843d2c98..95f265fdc 100644 --- a/openspec/changes/export-import-prompts-and-assistants/design.md +++ b/openspec/changes/export-import-prompts-and-assistants/design.md @@ -130,7 +130,7 @@ The LangSmith API has no documented way to run a prompt with its model, so a sho ### Developers commit the new SDK fields as `✨ feat(sdk)`, so the next release is a minor version -In [export-import-prompts-and-assistants Task group 3 SDK reads and writes the fields export and import need](tasks.md#L21), developers add public fields to `Prompt`, `CommitManifestResponse`, `CommitData`, `CommitRequest`, `Assistant`, `CreateAssistantRequest`, `UpdateAssistantRequest` and `AssistantSearchRequest`, and add new calls. They commit that work as `✨ feat(sdk)` with no `BREAKING CHANGE:` footer and no `!` after the type, and leave the version in `Cargo.toml` to the Prepare Release workflow. The repository's release docs ask for this: +Developers commit the SDK's new fields and calls as `✨ feat(sdk)`, with no `BREAKING CHANGE:` footer and no `!` after the type, and the Prepare Release workflow sets the version in `Cargo.toml`. The new fields go on `Prompt`, `CommitManifestResponse`, `CommitData`, `CommitRequest`, `Assistant`, `CreateAssistantRequest`, `UpdateAssistantRequest` and `AssistantSearchRequest`, in [export-import-prompts-and-assistants Task group 3 SDK reads and writes the fields export and import need](tasks.md#L21). The repository's release docs ask for this: - [docs/dev/git-scm-conventions.md:26](https://github.com/codekiln/langstar/blob/ab158b14160cf7803a79384f88756911bcd21857/docs/dev/git-scm-conventions.md#L26) gives `✨ feat` a minor version bump, and [docs/dev/git-scm-conventions.md:28](https://github.com/codekiln/langstar/blob/ab158b14160cf7803a79384f88756911bcd21857/docs/dev/git-scm-conventions.md#L28) keeps the major bump for `🚨 BREAKING CHANGE`. The table at [docs/dev/ci-cd.md:209](https://github.com/codekiln/langstar/blob/ab158b14160cf7803a79384f88756911bcd21857/docs/dev/ci-cd.md#L209) adds a `BREAKING CHANGE:` footer and `!` before the colon to the major bump. - [docs/dev/procedures.md:116](https://github.com/codekiln/langstar/blob/ab158b14160cf7803a79384f88756911bcd21857/docs/dev/procedures.md#L116) names changing function signatures and removing fields as breaking SDK changes. Adding a field or a call is neither. From cbaa75e19ff4d25b0d14e8765e9c5e97d9a6a5d8 Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Tue, 27 Oct 2026 16:45:01 -0400 Subject: [PATCH 72/90] =?UTF-8?q?=F0=9F=93=9A=20docs(openspec):=20flag=20p?= =?UTF-8?q?lural=20key=20names,=20redact=20the=20prompt=20reference=20erro?= =?UTF-8?q?r=20and=20support=20one=20import=20at=20a=20time?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Match `keys` as well as `key` after the credential prefixes so `api_keys` is flagged as task 4.4 tests, print `` for a flagged value in the prompt reference mismatch error, and say that langstar supports one import at a time per deployment, with #828 for making concurrent imports safe. Also retitle the verify decision and give the ci-cd.md sentence a subject that can act. Co-Authored-By: Claude Opus 5.5 --- .../export-import-prompts-and-assistants/design.md | 8 ++++---- .../specs/assistant/spec.md | 11 +++++++++-- .../specs/prompt/spec.md | 2 +- .../export-import-prompts-and-assistants/tasks.md | 2 +- 4 files changed, 15 insertions(+), 8 deletions(-) diff --git a/openspec/changes/export-import-prompts-and-assistants/design.md b/openspec/changes/export-import-prompts-and-assistants/design.md index 95f265fdc..5d247ec40 100644 --- a/openspec/changes/export-import-prompts-and-assistants/design.md +++ b/openspec/changes/export-import-prompts-and-assistants/design.md @@ -61,7 +61,7 @@ Before it writes anything, export runs a credential check over every string it w The check flags a string when the name of its key or the shape of its value marks it as a credential. -- **By key name:** the check flags a string whose key, or the key of the array it sits in, holds one of the words `secret`, `token`, `password`, `passwd`, `pwd`, `credential`, `credentials`, `apikey`, `authorization` or `cookie`, or holds `key` right after `api`, `access`, `private`, `secret`, `signing` or `account`. It splits a key name into words at `_`, `-`, `.`, spaces and each change from a lower-case to an upper-case letter, and ignores case. It flags `openai_api_key`, `OPENAI_API_KEY`, `x-api-key`, `Authorization`, `access_token`, `client_secret`, `db_password` and `privateKey`, and passes `max_tokens` and `cache_key`. +- **By key name:** the check flags a string whose key, or the key of the array it sits in, holds one of the words `secret`, `token`, `password`, `passwd`, `pwd`, `credential`, `credentials`, `apikey`, `authorization` or `cookie`, or holds `key` or `keys` right after `api`, `access`, `private`, `secret`, `signing` or `account`. It splits a key name into words at `_`, `-`, `.`, spaces and each change from a lower-case to an upper-case letter, and ignores case. It flags `openai_api_key`, `OPENAI_API_KEY`, `x-api-key`, `Authorization`, `access_token`, `client_secret`, `db_password`, `privateKey` and `api_keys`, and passes `max_tokens` and `cache_key`. - **By value shape**, under any key: the check flags a string that holds a bearer token (`Bearer ` followed by at least 16 token characters) or a `Basic ` value followed by at least 16 base64 characters; a PEM private key (`-----BEGIN` and `PRIVATE KEY-----`); a URL with a password in it, `scheme://user:password@host` or `scheme://:password@host`, which covers database and queue connection strings; a key-value connection string with a non-empty `Password=`, `Pwd=`, `AccountKey=`, `SharedAccessKey=` or `SharedAccessSignature=`; a JSON Web Token, three base64url parts that start with `eyJ`; or a token with a provider's published prefix, which are `sk-` (OpenAI and Anthropic), `lsv2_` (LangSmith), `ghp_`, `gho_`, `ghs_` and `github_pat_` (GitHub), `xoxb-`, `xoxp-` and `xoxa-` (Slack), `AKIA` and `ASIA` followed by 16 upper-case letters or digits (AWS), `AIza` (Google) and `hf_` (Hugging Face), each followed by enough characters to rule out an ordinary word. When the check flags any string, export writes no file and nothing to stdout, lists the JSON Pointer path of each flagged string with the rule that flagged it, such as `key name` or `URL with a password`, leaves the string and every part of it out of the message, and exits with status 1. Export refuses instead of writing `` in place of the credential, because a redacted file would import a prompt or assistant that cannot authenticate, and would hide from a reviewer reading the file in a pull request that the source holds a credential. The refusal asks the user to move the credential into the workspace's secrets, which a prompt's model reads through a secret reference, or into the deployment's environment variables, which a graph reads. @@ -76,7 +76,7 @@ Export reads the commit with `include=model`, the form the Go SDK uses (`referen ### Import records what it created in a separate file -Import writes the IDs of what it created to `.destinations.json`, keyed by destination workspace ID for prompts and by deployment ID for assistants. The export file stays the same after any number of imports, and a fresh export never wipes out the record of earlier imports. Import replaces the record by writing the whole new record to a temporary file in the same directory and renaming it over the old one, so a langstar that stops in the middle of a write leaves the earlier record whole, pending assistant IDs included. +Import writes the IDs of what it created to `.destinations.json`, keyed by destination workspace ID for prompts and by deployment ID for assistants. The export file stays the same after any number of imports, and a fresh export never wipes out the record of earlier imports. Import replaces the record by writing the whole new record to a temporary file in the same directory and renaming it over the old one, so a langstar that stops in the middle of a write leaves the earlier record whole, pending assistant IDs included. Langstar supports one import at a time per deployment, because a person runs import by hand from a Git checkout; two imports of the same file at once can overwrite each other's record entries, and [#828 make concurrent prompt and assistant imports into one deployment safe](https://github.com/codekiln/langstar/issues/828) covers making that safe. A repeated prompt import finds the copy by its handle and uses the record to tell whether someone changed it. Import writes the new commit hash to the record before it reads the commit back, so a read-back that finds a difference still leaves the hash in the record. The next import compares the destination's latest commit with that hash, and reports a change only when someone else pushed a commit after langstar's. Langstar can stop after it pushes a commit and before it writes that commit's hash to the record. The next import finds the destination's latest commit matching the file, writes its hash to the record and reports the prompt unchanged. @@ -98,7 +98,7 @@ Without `--update`, import lists the differences and exits with status 3. `--dry Compare reads the prompt or assistant from the destination, puts it in the shape of the export file's `repo`, `commit` and `assistant` objects, and lists every value that differs from the file. Compare checks every key in those objects, including the chat model's settings and the `id` arrays inside a manifest that name the prompt and model classes, such as `ChatOpenAI`. It skips `commit.model_config` and `commit.model_provider`, because LangSmith fills both in itself when import pushes a commit, and a destination workspace can fill them in differently from the source. Export leaves the values the server sets, such as IDs, version numbers, timestamps, owner and like counts, out of those objects, so they never show up as differences. The source environment's workspace, deployment and IDs sit in `source`, so they never show up as differences against a copy. Each difference names its JSON Pointer path and both values. Before the report prints either side of a difference, it replaces with `` every string that export's credential check flags, by either rule, whether the difference is at that string or at an object that contains it. A credential someone pasted into the destination prompt or the destination assistant, or into the file, therefore stays out of the terminal and the CI log. The report prints a LangChain secret reference, such as `{"lc": 1, "type": "secret", "id": ["OPENAI_API_KEY"]}`, unchanged, because the reference holds only the secret's name, never its value. The report prints to a terminal or a CI log, where the user needs to see where each difference is, so it redacts the string and goes on reporting. For a prompt, the comparison covers the commit description, the commit tags and the commit each one points at, any tag on the compared commit that the file lacks, and the repo's description, readme, tags and visibility as well as the manifest. Import runs the same comparison after it writes, so a field LangSmith or the Agent Server drops shows up as a difference. -### Verify checks what the copy needs to run +### `prompt verify` and `assistant verify` check the destination's secrets, graph and prompts Prompt verify checks that the workspace holds every secret the prompt's model reads. Export lists those names in `requires.secrets` by walking the manifest for LangChain secret references, `{"lc": 1, "type": "secret", "id": ["OPENAI_API_KEY"]}`, the shape the Python SDK's `_manifest_has_secrets` matches. Verify looks each name up among the workspace's secrets (`GET /api/v1/workspaces/current/secrets`, which `langstar secrets list` already calls) and prints the name of each missing secret and no secret value. @@ -132,7 +132,7 @@ The LangSmith API has no documented way to run a prompt with its model, so a sho Developers commit the SDK's new fields and calls as `✨ feat(sdk)`, with no `BREAKING CHANGE:` footer and no `!` after the type, and the Prepare Release workflow sets the version in `Cargo.toml`. The new fields go on `Prompt`, `CommitManifestResponse`, `CommitData`, `CommitRequest`, `Assistant`, `CreateAssistantRequest`, `UpdateAssistantRequest` and `AssistantSearchRequest`, in [export-import-prompts-and-assistants Task group 3 SDK reads and writes the fields export and import need](tasks.md#L21). The repository's release docs ask for this: -- [docs/dev/git-scm-conventions.md:26](https://github.com/codekiln/langstar/blob/ab158b14160cf7803a79384f88756911bcd21857/docs/dev/git-scm-conventions.md#L26) gives `✨ feat` a minor version bump, and [docs/dev/git-scm-conventions.md:28](https://github.com/codekiln/langstar/blob/ab158b14160cf7803a79384f88756911bcd21857/docs/dev/git-scm-conventions.md#L28) keeps the major bump for `🚨 BREAKING CHANGE`. The table at [docs/dev/ci-cd.md:209](https://github.com/codekiln/langstar/blob/ab158b14160cf7803a79384f88756911bcd21857/docs/dev/ci-cd.md#L209) adds a `BREAKING CHANGE:` footer and `!` before the colon to the major bump. +- [docs/dev/git-scm-conventions.md:26](https://github.com/codekiln/langstar/blob/ab158b14160cf7803a79384f88756911bcd21857/docs/dev/git-scm-conventions.md#L26) gives `✨ feat` a minor version bump, and [docs/dev/git-scm-conventions.md:28](https://github.com/codekiln/langstar/blob/ab158b14160cf7803a79384f88756911bcd21857/docs/dev/git-scm-conventions.md#L28) keeps the major bump for `🚨 BREAKING CHANGE`. [docs/dev/ci-cd.md:209](https://github.com/codekiln/langstar/blob/ab158b14160cf7803a79384f88756911bcd21857/docs/dev/ci-cd.md#L209) lists a `BREAKING CHANGE:` footer and a `!` before the colon as two more signs of a major bump. - [docs/dev/procedures.md:116](https://github.com/codekiln/langstar/blob/ab158b14160cf7803a79384f88756911bcd21857/docs/dev/procedures.md#L116) names changing function signatures and removing fields as breaking SDK changes. Adding a field or a call is neither. - The Prepare Release workflow, [.github/workflows/prepare-release.yml:62](https://github.com/codekiln/langstar/blob/ab158b14160cf7803a79384f88756911bcd21857/.github/workflows/prepare-release.yml#L62), picks the bump by running the bump-release skill's `analyze_commits.py` over the commits since the last tag, and writes the new version itself. git-cliff writes the changelog from the same commit messages, so nobody writes release notes or a version by hand. - [release.toml:5](https://github.com/codekiln/langstar/blob/ab158b14160cf7803a79384f88756911bcd21857/release.toml#L5) sets `publish = false`, which keeps `langstar-sdk` off crates.io, so the only crate known to build these structs is the CLI in this repository. diff --git a/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md b/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md index dfb088c5f..faf7c5074 100644 --- a/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md +++ b/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md @@ -39,7 +39,7 @@ Before it writes, `langstar assistant export` SHALL run the credential check ove ### Requirement: Flag an assistant string by the name of its key -The credential check SHALL flag a string when the key that holds it, or the array it is in, has a name whose words, split at `_`, `-`, `.`, spaces and lower-to-upper case changes and compared without regard to case, include `secret`, `token`, `password`, `passwd`, `pwd`, `credential`, `credentials`, `apikey`, `authorization` or `cookie`, or `key` right after `api`, `access`, `private`, `secret`, `signing` or `account`. +The credential check SHALL flag a string when the key that holds it, or the array it is in, has a name whose words, split at `_`, `-`, `.`, spaces and lower-to-upper case changes and compared without regard to case, include `secret`, `token`, `password`, `passwd`, `pwd`, `credential`, `credentials`, `apikey`, `authorization` or `cookie`, or `key` or `keys` right after `api`, `access`, `private`, `secret`, `signing` or `account`. #### Scenario: Context holds an access token @@ -260,7 +260,7 @@ After writing, `langstar assistant import` SHALL record the assistant ID in the ### Requirement: Reject a file whose prompt reference no longer matches its assistant -`assistant import`, `assistant compare` and `assistant verify` SHALL resolve each pointer under `requires.prompts` against the file's `assistant` object when they read the file. When a pointer resolves to no string, or to a string other than the prompt name recorded with it, they SHALL write nothing, exit with status 1 and name the pointer and both values. +`assistant import`, `assistant compare` and `assistant verify` SHALL resolve each pointer under `requires.prompts` against the file's `assistant` object when they read the file. When a pointer resolves to no string, or to a string other than the prompt name recorded with it, they SHALL write nothing, exit with status 1 and name the pointer and both values, with `` in place of each value the credential check flags, as the difference report does. #### Scenario: Configuration edited after export @@ -269,6 +269,13 @@ After writing, `langstar assistant import` SHALL record the assistant ID in the - **AND** the user runs `langstar assistant verify support-bot.json --deployment my-agent-prod` - **THEN** langstar exits with status 1 and names the pointer, `support-system:prod` and `support-system:staging` +#### Scenario: Edited prompt reference holds an API key + +- **WHEN** `requires.prompts` records `/config/configurable/system_prompt` with `support-system:prod` +- **AND** the file's `assistant.config.configurable.system_prompt` holds a string that starts with `sk-` +- **AND** the user runs `langstar assistant verify support-bot.json --deployment my-agent-prod` +- **THEN** langstar exits with status 1, names the pointer and `support-system:prod`, and prints `` in place of the string + ### Requirement: Assistant compare, verify and import use the prompt commands' exit statuses and JSON report `assistant compare`, `assistant verify` and `assistant import` SHALL use the exit statuses and the JSON report shape of `prompt compare`, `prompt verify` and `prompt import`. diff --git a/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md index a27dbca77..a4490380d 100644 --- a/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md +++ b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md @@ -96,7 +96,7 @@ Before it writes, `langstar prompt export` SHALL run the credential check over e ### Requirement: Flag a prompt string by the name of its key -The credential check SHALL flag a string when the key that holds it, or the array it is in, has a name whose words, split at `_`, `-`, `.`, spaces and lower-to-upper case changes and compared without regard to case, include `secret`, `token`, `password`, `passwd`, `pwd`, `credential`, `credentials`, `apikey`, `authorization` or `cookie`, or `key` right after `api`, `access`, `private`, `secret`, `signing` or `account`. +The credential check SHALL flag a string when the key that holds it, or the array it is in, has a name whose words, split at `_`, `-`, `.`, spaces and lower-to-upper case changes and compared without regard to case, include `secret`, `token`, `password`, `passwd`, `pwd`, `credential`, `credentials`, `apikey`, `authorization` or `cookie`, or `key` or `keys` right after `api`, `access`, `private`, `secret`, `signing` or `account`. #### Scenario: Model sends an authorization header diff --git a/openspec/changes/export-import-prompts-and-assistants/tasks.md b/openspec/changes/export-import-prompts-and-assistants/tasks.md index 7d430b772..915ff5c10 100644 --- a/openspec/changes/export-import-prompts-and-assistants/tasks.md +++ b/openspec/changes/export-import-prompts-and-assistants/tasks.md @@ -130,7 +130,7 @@ Each integration test in this group carries `#[cfg_attr(not(feature = "integrati - An integration test that exports an assistant, changes its metadata through the SDK, runs compare with `--assistant-id` set to that assistant, and asserts the reported path and exit status 3. - Unit tests that compare reads the assistant ID from a destination record written by the 4.1 types, once at the default path and once at a path passed with `--record`. - An integration test that `assistant compare` with no destination record entry and no `--assistant-id` exits with status 1 and names the deployment. - - A unit test that a file whose configuration names another prompt at a pointer under `requires.prompts` exits with status 1 and names the pointer and both values. + - A unit test that a file whose configuration names another prompt at a pointer under `requires.prompts` exits with status 1 and names the pointer and both values, and one whose configuration holds a string that starts with `sk-` at that pointer, which prints `` in place of the string. - [ ] 9.2 Add `langstar assistant verify` with `--assistant-id`, `--record`, `--workspace-id` for the workspace that holds the prompts, the graph check, and the prompt check, which reads `GET /api/v1/commits/{owner}/{repo}/{tag}`. Verify with these tests: - An integration test that runs verify with `--assistant-id` against an exported assistant and exits with status 0. - An integration test that exports an assistant, changes its metadata through the SDK, runs verify with `--assistant-id` set to that assistant, and asserts the reported path and exit status 3. From df2242d1e77eca2f3038aaa2b852b5080b399782 Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Tue, 27 Oct 2026 16:59:37 -0400 Subject: [PATCH 73/90] =?UTF-8?q?=F0=9F=93=9A=20docs(openspec):=20name=20w?= =?UTF-8?q?hat=20one-import-at-a-time=20protects=20and=20untangle=20the=20?= =?UTF-8?q?credential=20sentences?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Say that langstar supports one import of an export file at a time, what two concurrent imports do to the record and to a first assistant import, and link #828. Split the redaction sentence whose object came last, use "lets through" for the check and "names" for the user, give the sk- scenarios a token long enough to flag, drop the "two more signs" count, and stop the proposal's link text from repeating its sentence. Co-Authored-By: Claude Opus 5.5 --- .../export-import-prompts-and-assistants/design.md | 8 ++++---- .../export-import-prompts-and-assistants/proposal.md | 2 +- .../specs/assistant/spec.md | 4 ++-- .../specs/prompt/spec.md | 6 +++--- .../changes/export-import-prompts-and-assistants/tasks.md | 2 +- 5 files changed, 11 insertions(+), 11 deletions(-) diff --git a/openspec/changes/export-import-prompts-and-assistants/design.md b/openspec/changes/export-import-prompts-and-assistants/design.md index 5d247ec40..01ea8cbc4 100644 --- a/openspec/changes/export-import-prompts-and-assistants/design.md +++ b/openspec/changes/export-import-prompts-and-assistants/design.md @@ -64,9 +64,9 @@ The check flags a string when the name of its key or the shape of its value mark - **By key name:** the check flags a string whose key, or the key of the array it sits in, holds one of the words `secret`, `token`, `password`, `passwd`, `pwd`, `credential`, `credentials`, `apikey`, `authorization` or `cookie`, or holds `key` or `keys` right after `api`, `access`, `private`, `secret`, `signing` or `account`. It splits a key name into words at `_`, `-`, `.`, spaces and each change from a lower-case to an upper-case letter, and ignores case. It flags `openai_api_key`, `OPENAI_API_KEY`, `x-api-key`, `Authorization`, `access_token`, `client_secret`, `db_password`, `privateKey` and `api_keys`, and passes `max_tokens` and `cache_key`. - **By value shape**, under any key: the check flags a string that holds a bearer token (`Bearer ` followed by at least 16 token characters) or a `Basic ` value followed by at least 16 base64 characters; a PEM private key (`-----BEGIN` and `PRIVATE KEY-----`); a URL with a password in it, `scheme://user:password@host` or `scheme://:password@host`, which covers database and queue connection strings; a key-value connection string with a non-empty `Password=`, `Pwd=`, `AccountKey=`, `SharedAccessKey=` or `SharedAccessSignature=`; a JSON Web Token, three base64url parts that start with `eyJ`; or a token with a provider's published prefix, which are `sk-` (OpenAI and Anthropic), `lsv2_` (LangSmith), `ghp_`, `gho_`, `ghs_` and `github_pat_` (GitHub), `xoxb-`, `xoxp-` and `xoxa-` (Slack), `AKIA` and `ASIA` followed by 16 upper-case letters or digits (AWS), `AIza` (Google) and `hf_` (Hugging Face), each followed by enough characters to rule out an ordinary word. -When the check flags any string, export writes no file and nothing to stdout, lists the JSON Pointer path of each flagged string with the rule that flagged it, such as `key name` or `URL with a password`, leaves the string and every part of it out of the message, and exits with status 1. Export refuses instead of writing `` in place of the credential, because a redacted file would import a prompt or assistant that cannot authenticate, and would hide from a reviewer reading the file in a pull request that the source holds a credential. The refusal asks the user to move the credential into the workspace's secrets, which a prompt's model reads through a secret reference, or into the deployment's environment variables, which a graph reads. +When the check flags any string, export writes no file and nothing to stdout, lists the JSON Pointer path of each flagged string with the rule that flagged it, such as `key name` or `URL with a password`, leaves the string and every part of it out of the message, and exits with status 1. Export refuses rather than writing `` in place of the credential. A redacted file would import a prompt or assistant that cannot authenticate, and a reviewer reading the file in a pull request would not see that the source holds a credential. The refusal asks the user to move the credential into the workspace's secrets, which a prompt's model reads through a secret reference, or into the deployment's environment variables, which a graph reads. -Some flagged strings are harmless, such as a message template that shows `Bearer ` followed by an example token. A user who has read such a string passes `--not-a-secret `, once for each path, and export writes the string at that path and prints a warning to stderr naming the path. When a pointer names a string the check passes, export refuses as well: it names the pointer and exits with status 1, so a script that still passes the pointer after the string changed stops at the export. Every export runs the check. +Some flagged strings are harmless, such as a message template that shows `Bearer ` followed by an example token. A user who has read such a string names its path with `--not-a-secret `, once for each path, and export writes the string and prints a warning to stderr naming the path. When a pointer names a string the check lets through, export writes nothing, names the pointer and exits with status 1, so a script that keeps naming a path after the string there has changed stops at the export. Every export runs the check. A credential under an ordinary key name with an ordinary-looking value, such as a random string under a key called `x`, passes the check and reaches the file. The export docs tell users to keep credentials in secrets and environment variables, and to read an export file before committing it. @@ -76,7 +76,7 @@ Export reads the commit with `include=model`, the form the Go SDK uses (`referen ### Import records what it created in a separate file -Import writes the IDs of what it created to `.destinations.json`, keyed by destination workspace ID for prompts and by deployment ID for assistants. The export file stays the same after any number of imports, and a fresh export never wipes out the record of earlier imports. Import replaces the record by writing the whole new record to a temporary file in the same directory and renaming it over the old one, so a langstar that stops in the middle of a write leaves the earlier record whole, pending assistant IDs included. Langstar supports one import at a time per deployment, because a person runs import by hand from a Git checkout; two imports of the same file at once can overwrite each other's record entries, and [#828 make concurrent prompt and assistant imports into one deployment safe](https://github.com/codekiln/langstar/issues/828) covers making that safe. +Import writes the IDs of what it created to `.destinations.json`, keyed by destination workspace ID for prompts and by deployment ID for assistants. The export file stays the same after any number of imports, and a fresh export never wipes out the record of earlier imports. Import replaces the record by writing the whole new record to a temporary file in the same directory and renaming it over the old one, so a langstar that stops in the middle of a write leaves the earlier record whole, pending assistant IDs included. Langstar supports one import of an export file at a time, because a person runs import by hand from a Git checkout. When two imports of the same file run at once, each reads the record before the other writes it, and the import that writes last drops the other's entries. Two first imports of one assistant into the same deployment can also create two assistants. [#828 make concurrent prompt and assistant imports into one deployment safe](https://github.com/codekiln/langstar/issues/828) covers making concurrent imports safe. A repeated prompt import finds the copy by its handle and uses the record to tell whether someone changed it. Import writes the new commit hash to the record before it reads the commit back, so a read-back that finds a difference still leaves the hash in the record. The next import compares the destination's latest commit with that hash, and reports a change only when someone else pushed a commit after langstar's. Langstar can stop after it pushes a commit and before it writes that commit's hash to the record. The next import finds the destination's latest commit matching the file, writes its hash to the record and reports the prompt unchanged. @@ -132,7 +132,7 @@ The LangSmith API has no documented way to run a prompt with its model, so a sho Developers commit the SDK's new fields and calls as `✨ feat(sdk)`, with no `BREAKING CHANGE:` footer and no `!` after the type, and the Prepare Release workflow sets the version in `Cargo.toml`. The new fields go on `Prompt`, `CommitManifestResponse`, `CommitData`, `CommitRequest`, `Assistant`, `CreateAssistantRequest`, `UpdateAssistantRequest` and `AssistantSearchRequest`, in [export-import-prompts-and-assistants Task group 3 SDK reads and writes the fields export and import need](tasks.md#L21). The repository's release docs ask for this: -- [docs/dev/git-scm-conventions.md:26](https://github.com/codekiln/langstar/blob/ab158b14160cf7803a79384f88756911bcd21857/docs/dev/git-scm-conventions.md#L26) gives `✨ feat` a minor version bump, and [docs/dev/git-scm-conventions.md:28](https://github.com/codekiln/langstar/blob/ab158b14160cf7803a79384f88756911bcd21857/docs/dev/git-scm-conventions.md#L28) keeps the major bump for `🚨 BREAKING CHANGE`. [docs/dev/ci-cd.md:209](https://github.com/codekiln/langstar/blob/ab158b14160cf7803a79384f88756911bcd21857/docs/dev/ci-cd.md#L209) lists a `BREAKING CHANGE:` footer and a `!` before the colon as two more signs of a major bump. +- [docs/dev/git-scm-conventions.md:26](https://github.com/codekiln/langstar/blob/ab158b14160cf7803a79384f88756911bcd21857/docs/dev/git-scm-conventions.md#L26) gives `✨ feat` a minor version bump, and [docs/dev/git-scm-conventions.md:28](https://github.com/codekiln/langstar/blob/ab158b14160cf7803a79384f88756911bcd21857/docs/dev/git-scm-conventions.md#L28) keeps the major bump for `🚨 BREAKING CHANGE`. [docs/dev/ci-cd.md:209](https://github.com/codekiln/langstar/blob/ab158b14160cf7803a79384f88756911bcd21857/docs/dev/ci-cd.md#L209) also gives a major bump to a commit with a `BREAKING CHANGE:` footer or a `!` before the colon. - [docs/dev/procedures.md:116](https://github.com/codekiln/langstar/blob/ab158b14160cf7803a79384f88756911bcd21857/docs/dev/procedures.md#L116) names changing function signatures and removing fields as breaking SDK changes. Adding a field or a call is neither. - The Prepare Release workflow, [.github/workflows/prepare-release.yml:62](https://github.com/codekiln/langstar/blob/ab158b14160cf7803a79384f88756911bcd21857/.github/workflows/prepare-release.yml#L62), picks the bump by running the bump-release skill's `analyze_commits.py` over the commits since the last tag, and writes the new version itself. git-cliff writes the changelog from the same commit messages, so nobody writes release notes or a version by hand. - [release.toml:5](https://github.com/codekiln/langstar/blob/ab158b14160cf7803a79384f88756911bcd21857/release.toml#L5) sets `publish = false`, which keeps `langstar-sdk` off crates.io, so the only crate known to build these structs is the CLI in this repository. diff --git a/openspec/changes/export-import-prompts-and-assistants/proposal.md b/openspec/changes/export-import-prompts-and-assistants/proposal.md index cfb8b4a18..d16f04ad9 100644 --- a/openspec/changes/export-import-prompts-and-assistants/proposal.md +++ b/openspec/changes/export-import-prompts-and-assistants/proposal.md @@ -31,7 +31,7 @@ A prompt belongs to a workspace and an assistant belongs to a deployment, but bo ## Impact -- SDK: `sdk/src/prompts.rs` gains the prompt calls export and import need: a commit read that asks for the model and returns it on `CommitManifestResponse`, a commit create that sends a description and returns the new commit's `id`, a commit list, calls that list, create, move and delete commit tags, and a repo update. `Prompt` gains the repo's readme, tags and `tenant_id`, and export fills `source.workspace_id` from `tenant_id`. `sdk/src/assistants.rs` gains the `context` and `version` fields, `assistant_id` and `if_exists` on create, `graph_id` on update, and a search by graph ID. Developers commit these fields as `✨ feat(sdk)` with no `BREAKING CHANGE:` footer, so the Prepare Release workflow makes the next release a minor version; [Developers commit the new SDK fields as `✨ feat(sdk)`, so the next release is a minor version](design.md#developers-commit-the-new-sdk-fields-as--featsdk-so-the-next-release-is-a-minor-version) cites the release docs. +- SDK: `sdk/src/prompts.rs` gains the prompt calls export and import need: a commit read that asks for the model and returns it on `CommitManifestResponse`, a commit create that sends a description and returns the new commit's `id`, a commit list, calls that list, create, move and delete commit tags, and a repo update. `Prompt` gains the repo's readme, tags and `tenant_id`, and export fills `source.workspace_id` from `tenant_id`. `sdk/src/assistants.rs` gains the `context` and `version` fields, `assistant_id` and `if_exists` on create, `graph_id` on update, and a search by graph ID. Developers commit these fields as `✨ feat(sdk)` with no `BREAKING CHANGE:` footer, so the Prepare Release workflow makes the next release a minor version. [design.md's decision on the SDK release](design.md#developers-commit-the-new-sdk-fields-as--featsdk-so-the-next-release-is-a-minor-version) links the release docs that ask for this. - CLI: new subcommands in `cli/src/commands/prompt.rs` and `cli/src/commands/assistant.rs`, and a shared module for the export file, the destination record, comparison and the exit status. - Tests need a second workspace, a second deployment and one model provider key for the structured output check. codekiln creates the workspace and the key, and langstar's tests read them through `fnox.toml` and the CI workflow; the destination deployment fixture from [export-import-prompts-and-assistants Task 2.4 Add a destination deployment fixture](tasks.md#L19) creates the deployment whenever the cleanup workflow has deleted it, as the source fixture does today. Nobody knows yet whether the control plane accepts a second deployment from this repository. In [export-import-prompts-and-assistants Task 2.3 Register the test graph under a second graph ID](tasks.md#L18), a developer deploys the new `langgraph.json` once to find out, before anyone builds the fixture. - Docs: a new usage page on moving prompts and assistants between environments. diff --git a/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md b/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md index faf7c5074..ea3f5577a 100644 --- a/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md +++ b/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md @@ -77,7 +77,7 @@ The credential check SHALL flag a string, under any key, that holds a bearer tok ### Requirement: Write each flagged assistant string the user names with --not-a-secret -`langstar assistant export` SHALL take `--not-a-secret ` any number of times. For each pointer at a string the credential check flags, export SHALL write that string and print a warning to stderr naming the pointer. A flagged string at any path not named SHALL still make export refuse. When a pointer names a string the check passes, export SHALL write nothing, name the pointer and exit with status 1. Export SHALL run the check on every run. +`langstar assistant export` SHALL take `--not-a-secret ` any number of times. For each pointer at a string the credential check flags, export SHALL write that string and print a warning to stderr naming the pointer. A flagged string at any path not named SHALL still make export refuse. When a pointer names a string the check lets through, export SHALL write nothing, name the pointer and exit with status 1. Export SHALL run the check on every run. #### Scenario: User writes a token budget setting with --not-a-secret @@ -272,7 +272,7 @@ After writing, `langstar assistant import` SHALL record the assistant ID in the #### Scenario: Edited prompt reference holds an API key - **WHEN** `requires.prompts` records `/config/configurable/system_prompt` with `support-system:prod` -- **AND** the file's `assistant.config.configurable.system_prompt` holds a string that starts with `sk-` +- **AND** the file's `assistant.config.configurable.system_prompt` holds `sk-` followed by a 40-character token - **AND** the user runs `langstar assistant verify support-bot.json --deployment my-agent-prod` - **THEN** langstar exits with status 1, names the pointer and `support-system:prod`, and prints `` in place of the string diff --git a/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md index a4490380d..cfc63dfcf 100644 --- a/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md +++ b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md @@ -116,7 +116,7 @@ The credential check SHALL flag a string, under any key, that holds a bearer tok #### Scenario: Model setting holds a token under a neutral name -- **WHEN** the prompt's chat model holds a string that starts with `sk-` under `model_kwargs.extra` +- **WHEN** the prompt's chat model holds `sk-` followed by a 40-character token under `model_kwargs.extra` - **AND** a user runs `langstar prompt export my-prompt` - **THEN** stdout is empty, langstar names the path of `extra` with the rule that flagged it and exits with status 1 @@ -128,7 +128,7 @@ The credential check SHALL flag a string, under any key, that holds a bearer tok ### Requirement: Write each flagged prompt string the user names with --not-a-secret -`langstar prompt export` SHALL take `--not-a-secret ` any number of times. For each pointer at a string the credential check flags, export SHALL write that string and print a warning to stderr naming the pointer. A flagged string at any path not named SHALL still make export refuse. When a pointer names a string the check passes, export SHALL write nothing, name the pointer and exit with status 1. Export SHALL run the check on every run. +`langstar prompt export` SHALL take `--not-a-secret ` any number of times. For each pointer at a string the credential check flags, export SHALL write that string and print a warning to stderr naming the pointer. A flagged string at any path not named SHALL still make export refuse. When a pointer names a string the check lets through, export SHALL write nothing, name the pointer and exit with status 1. Export SHALL run the check on every run. #### Scenario: Message template shows an example token @@ -136,7 +136,7 @@ The credential check SHALL flag a string, under any key, that holds a bearer tok - **AND** a user runs `langstar prompt export my-prompt --output my-prompt.json --not-a-secret ` - **THEN** langstar writes the file with the template unchanged, prints a warning naming the path to stderr and exits with status 0 -#### Scenario: User passes --not-a-secret for a string the check passes +#### Scenario: User names an unflagged string with --not-a-secret - **WHEN** a user runs `langstar prompt export my-prompt --output my-prompt.json --not-a-secret /repo/description` - **AND** the check flags nothing at `/repo/description` diff --git a/openspec/changes/export-import-prompts-and-assistants/tasks.md b/openspec/changes/export-import-prompts-and-assistants/tasks.md index 915ff5c10..b9f83d2f4 100644 --- a/openspec/changes/export-import-prompts-and-assistants/tasks.md +++ b/openspec/changes/export-import-prompts-and-assistants/tasks.md @@ -130,7 +130,7 @@ Each integration test in this group carries `#[cfg_attr(not(feature = "integrati - An integration test that exports an assistant, changes its metadata through the SDK, runs compare with `--assistant-id` set to that assistant, and asserts the reported path and exit status 3. - Unit tests that compare reads the assistant ID from a destination record written by the 4.1 types, once at the default path and once at a path passed with `--record`. - An integration test that `assistant compare` with no destination record entry and no `--assistant-id` exits with status 1 and names the deployment. - - A unit test that a file whose configuration names another prompt at a pointer under `requires.prompts` exits with status 1 and names the pointer and both values, and one whose configuration holds a string that starts with `sk-` at that pointer, which prints `` in place of the string. + - A unit test that a file whose configuration names another prompt at a pointer under `requires.prompts` exits with status 1 and names the pointer and both values. Another unit test puts `sk-` followed by a 40-character token at that pointer and asserts that the error prints `` in place of the string. - [ ] 9.2 Add `langstar assistant verify` with `--assistant-id`, `--record`, `--workspace-id` for the workspace that holds the prompts, the graph check, and the prompt check, which reads `GET /api/v1/commits/{owner}/{repo}/{tag}`. Verify with these tests: - An integration test that runs verify with `--assistant-id` against an exported assistant and exits with status 0. - An integration test that exports an assistant, changes its metadata through the SDK, runs verify with `--assistant-id` set to that assistant, and asserts the reported path and exit status 3. From b058131719de0898776a4424db4471a831f0799a Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Tue, 27 Oct 2026 17:10:55 -0400 Subject: [PATCH 74/90] =?UTF-8?q?=F0=9F=93=9A=20docs(openspec):=20define?= =?UTF-8?q?=20prompt-reference=20syntax,=20unknown-graph=20handling=20and?= =?UTF-8?q?=20assistant=20import=20without=20--update?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Copilot listed these as previously missed findings on #816: - Prompt names under requires.prompts take [/][:]; no owner means the current workspace, no tag means latest. - Verify counts a 422 naming graph_id from the assistant search as a missing graph; task 1.2 records what the live server returns. - Assistant import creates an assistant only on a first import, and --assistant-id without --update compares and changes nothing. - A separate requirement covers --record for assistant import, compare and verify. Co-Authored-By: Claude Opus 5.5 --- .../design.md | 2 +- .../specs/assistant/spec.md | 30 +++++++++++++++++-- .../tasks.md | 6 ++-- 3 files changed, 33 insertions(+), 5 deletions(-) diff --git a/openspec/changes/export-import-prompts-and-assistants/design.md b/openspec/changes/export-import-prompts-and-assistants/design.md index 01ea8cbc4..d294ef423 100644 --- a/openspec/changes/export-import-prompts-and-assistants/design.md +++ b/openspec/changes/export-import-prompts-and-assistants/design.md @@ -106,7 +106,7 @@ Prompt verify also notes when the workspace has no saved model configuration for Import and verify work out `requires` again from the file's manifest: the secret names, and the model class or null. When either differs from what the file holds, they write nothing, name the field and exit with status 1. This catches a manifest someone edited in Git without updating `requires`. -Assistant verify checks the graph with `POST /assistants/search` filtered to the graph ID; every graph on a deployment has a default assistant, so an empty result means the deployment does not serve the graph. It checks each prompt in `requires.prompts`, such as `support-system:prod`, by asking LangSmith for the commit that tag points at (`GET /api/v1/commits/{owner}/{repo}/{tag}`, the request `langstar prompt pull --commit prod` already sends). A 404 means the workspace lacks the prompt or its tag. +Assistant verify checks the graph with `POST /assistants/search` filtered to the graph ID; every graph on a deployment has a default assistant, so an empty result means the deployment does not serve the graph. The Agent Server's OpenAPI spec lists the deployment's graph IDs as the allowed values of the search's `graph_id`, so the server may answer an unknown graph ID with 422 instead of an empty result. Verify therefore also counts a 422 whose error names `graph_id` as a missing graph, and exits with status 1 on any other error; in [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8) a developer records which answer the live server gives. It checks each prompt in `requires.prompts`, such as `support-system:prod`, by asking LangSmith for the commit that tag points at (`GET /api/v1/commits/{owner}/{repo}/{tag}`, the request `langstar prompt pull --commit prod` already sends). A prompt name has the form `[/][:]`: without an owner verify uses `-`, the current workspace, and without a tag it asks for `latest`. A 404 means the workspace lacks the prompt or its tag. ### Import checks the destination's secrets, graph and prompts before it writes diff --git a/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md b/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md index ea3f5577a..c5733b2f7 100644 --- a/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md +++ b/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md @@ -87,7 +87,7 @@ The credential check SHALL flag a string, under any key, that holds a bearer tok ### Requirement: Record the prompts an assistant refers to -`langstar assistant export` SHALL accept `--prompt-ref `, repeatable, naming a string in the assistant's configuration or context that holds a prompt name. The file SHALL list each pointer and the prompt name it holds under `requires.prompts`. A pointer that resolves to no string SHALL be an error. +`langstar assistant export` SHALL accept `--prompt-ref `, repeatable, naming a string in the assistant's configuration or context that holds a prompt name of the form `[/][:]`. A name without an owner SHALL mean the current workspace's prompt, and a name without a tag SHALL mean the prompt's latest commit. The file SHALL list each pointer and the prompt name it holds under `requires.prompts`. A pointer that resolves to no string SHALL be an error. #### Scenario: Assistant names its system prompt @@ -95,6 +95,13 @@ The credential check SHALL flag a string, under any key, that holds a bearer tok - **AND** that value is `support-system:prod` - **THEN** `requires.prompts` holds the pointer and `support-system:prod` +#### Scenario: Assistant names a prompt without a tag + +- **WHEN** a user runs `langstar assistant export --deployment my-agent --prompt-ref /config/configurable/system_prompt` +- **AND** that value is `support-system` +- **THEN** `requires.prompts` holds the pointer and `support-system` +- **AND** `langstar assistant verify` checks the latest commit of the workspace's `support-system` prompt + #### Scenario: Pointer names nothing - **WHEN** the pointer resolves to no value in the assistant @@ -102,7 +109,7 @@ The credential check SHALL flag a string, under any key, that holds a bearer tok ### Requirement: Import an assistant into a deployment -`langstar assistant import --deployment ` SHALL choose a new assistant ID, write it as pending under the deployment in the destination record (`.destinations.json` unless `--record ` names another), then create the assistant with that `assistant_id`, `if_exists: do_nothing` and the file's name, description, graph ID, configuration, context and metadata, mark the entry created, and print the ID. A pending entry SHALL make the next import repeat the create. +With no created assistant ID recorded for the deployment and no `--assistant-id`, `langstar assistant import --deployment ` SHALL choose a new assistant ID, write it as pending in the destination record, then create the assistant with that `assistant_id`, `if_exists: do_nothing` and the file's name, description, graph ID, configuration, context and metadata, mark the entry created, and print the ID. A pending entry SHALL make the next import repeat the create. #### Scenario: First import @@ -119,6 +126,15 @@ The credential check SHALL flag a string, under any key, that holds a bearer tok - **THEN** langstar sends the create with that ID and `if_exists` set to `do_nothing`, so the deployment still holds one assistant from the file - **AND** the destination record marks that ID created +### Requirement: Keep the assistant destination record at the path the user names + +`langstar assistant import`, `assistant compare` and `assistant verify` SHALL read and write the destination record at `.destinations.json` next to the export file, unless `--record ` names another file. + +#### Scenario: Import writes the record to another path + +- **WHEN** a user runs `langstar assistant import support-bot.json --deployment my-agent-prod --record records/prod.json` +- **THEN** langstar writes the new assistant ID to `records/prod.json` and writes no `support-bot.destinations.json` + ### Requirement: Replace the destination record in one step `langstar assistant import` SHALL write each new destination record to a temporary file in the record's directory and rename it over the old record, so that a langstar that stops during a write leaves the earlier record whole, with its pending assistant IDs. @@ -169,6 +185,16 @@ When the destination record holds a created assistant ID for the deployment, `la - **WHEN** the user runs `langstar assistant import support-bot.json --deployment my-agent-prod --update --assistant-id ` - **THEN** langstar updates that assistant and records its ID for `my-agent-prod` +### Requirement: Compare with the assistant named by --assistant-id when --update is absent + +Without `--update`, `langstar assistant import --assistant-id ` SHALL compare the file with that assistant, as `assistant compare` does, and create and change nothing. It SHALL exit with status 0 when they match, and list the differences and exit with status 3 when they differ. + +#### Scenario: Named assistant differs from the file + +- **WHEN** the assistant `` has configuration that differs from the file +- **AND** the user runs `langstar assistant import support-bot.json --deployment my-agent-prod --assistant-id ` +- **THEN** langstar creates no assistant, changes none, lists the differences and exits with status 3 + ### Requirement: Import with --update asks the Agent Server to remove the metadata, description and context the file lacks `langstar assistant import --update` SHALL ask the Agent Server to remove each metadata key the destination assistant has and the file lacks, and to clear a description or context the file lacks, then read the assistant back. When such a key or field remains after the update, it SHALL list it as a difference and exit with status 3. diff --git a/openspec/changes/export-import-prompts-and-assistants/tasks.md b/openspec/changes/export-import-prompts-and-assistants/tasks.md index b9f83d2f4..37cd31ce8 100644 --- a/openspec/changes/export-import-prompts-and-assistants/tasks.md +++ b/openspec/changes/export-import-prompts-and-assistants/tasks.md @@ -8,7 +8,7 @@ Each numbered group below is one pull request, merged in the order listed, excep - [ ] 1.2 Write `reference/research/807-openapi-validation.md`, recording what the live API returns. Verify that every claim in the report names the request that produced it. - For a structured prompt saved with a model, read with `include=model`: the manifest, its outer `id`, the `model_config` and `model_provider` beside it, and whether `model_provider` matches the class name export writes to `requires.model_class`. - Push that manifest to a second workspace: whether the read-back returns the same manifest, `model_config` and `model_provider`, and what pushing an identical manifest again returns. - - On the Agent Server: what it returns for `context`, whether an assistant update that sends a metadata key with a null value removes that key, whether an update that sends `config` or `context` replaces the assistant's object or merges into it, and whether an update can clear the description. + - On the Agent Server: what `POST /assistants/search` returns for a graph ID the deployment does not serve, what it returns for `context`, whether an assistant update that sends a metadata key with a null value removes that key, whether an update that sends `config` or `context` replaces the assistant's object or merges into it, and whether an update can clear the description. - What `POST /api/v1/prompts/invoke_prompt` returns for a prompt with a model. ## 2. Test environments @@ -137,7 +137,9 @@ Each integration test in this group carries `#[cfg_attr(not(feature = "integrati - An integration test of a file whose graph ID the deployment lacks, which reports the graph ID and exits with status 3. - An integration test of a file whose prompt tag the workspace lacks, which reports the prompt and exits with status 3. - An integration test that runs verify with `--workspace-id` set to the destination workspace, for an assistant whose referenced prompt tag exists only in that workspace, and asserts that verify exits with status 0 and that `checks` reports the prompt as found. + - Unit tests in `cli/src/commands/assistant.rs` that serve, through a `mockito` server, a search that answers 422 with an error naming `graph_id`, and assert that verify reports the graph ID missing and exits with status 3, and a 422 naming another field, and assert that verify exits with status 1. - A CLI test that `langstar -f json assistant verify` prints one JSON object with `status`, `differences` and `checks` to stdout. + - A unit test that a prompt reference with no owner and no tag, such as `support-system`, makes the prompt check request `GET /api/v1/commits/-/support-system/latest`, and one that `my-org/support-system:prod` requests `GET /api/v1/commits/my-org/support-system/prod`. - [ ] 9.3 Document assistant compare and verify; verify that each documented command runs as written ## 10. assistant import @@ -152,7 +154,7 @@ Each integration test in this group carries `#[cfg_attr(not(feature = "integrati - [ ] 10.2 Add `--update`, `--assistant-id` and `--dry-run`. Verify with these tests: - An integration test that a repeated import creates no assistant. - An integration test that `--update` changes the recorded assistant, moving it from `test_graph` to `test_graph_alt` among the fields. - - An integration test that `--update --assistant-id ` naming an assistant other than the recorded one updates that assistant, leaves the recorded one unchanged and writes the new ID to the destination record. + - An integration test that `--update --assistant-id ` naming an assistant other than the recorded one updates that assistant, leaves the recorded one unchanged and writes the new ID to the destination record, and that `--assistant-id ` without `--update`, against an assistant that differs from the file, creates and changes no assistant and exits with status 3. - An integration test that `--update` of an assistant with metadata `{"kept": 1, "stale": 2}` from a file with `{"kept": 1}` leaves `{"kept": 1}`, or reports `/metadata/stale` and exits with status 3, whichever task 1.2 found the server does. - An integration test that `--update` of an assistant with a description and context from a file with neither clears both, or reports `/description` and `/context` and exits with status 3, whichever task 1.2 found the server does. - An integration test that a deleted recorded assistant exits with status 1. From 898cf53f876f95c334eb983e89c084ec91efb736 Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Tue, 27 Oct 2026 17:10:56 -0400 Subject: [PATCH 75/90] =?UTF-8?q?=F0=9F=93=9A=20docs(openspec):=20report?= =?UTF-8?q?=20import's=20action=20in=20JSON,=20settle=20a=20resumed=20impo?= =?UTF-8?q?rt's=20repo=20differences=20and=20pin=20the=20structured=20outp?= =?UTF-8?q?ut=20check?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Copilot listed these as previously missed findings on #816: - prompt import and assistant import JSON reports carry action and the destination's commit hash or assistant ID. - A resumed first import pushes the commit and tags, leaves differing repo details alone and exits with status 3 without --update. - The structured output script pins its dependencies and validates with jsonschema. - Task 1.2 waits on the destination workspace and key from task 2.1. Co-Authored-By: Claude Opus 5.5 --- .../export-import-prompts-and-assistants/design.md | 2 +- .../specs/prompt/spec.md | 12 +++++++++++- .../export-import-prompts-and-assistants/tasks.md | 10 ++++++---- 3 files changed, 18 insertions(+), 6 deletions(-) diff --git a/openspec/changes/export-import-prompts-and-assistants/design.md b/openspec/changes/export-import-prompts-and-assistants/design.md index d294ef423..fb5543e2d 100644 --- a/openspec/changes/export-import-prompts-and-assistants/design.md +++ b/openspec/changes/export-import-prompts-and-assistants/design.md @@ -88,7 +88,7 @@ A prompt that exists in the destination gets a new commit only with `--update`, Import treats the destination commit's tags as one more field to match. A tag in the file that the destination lacks, or that points at another commit there, is a difference, and so is a tag on the destination's matching commit that the file lacks. With `--update`, import creates or moves each of the file's tags to that commit and deletes each tag on it that the file lacks, through `DELETE /api/v1/repos/{owner}/{repo}/tags/{tag_name}`, so the read-back finds the same tags as the file. It adds no commit when the latest commit already matches. A tag on an older destination commit is not part of the comparison, so import leaves it alone. The tag endpoints take a commit's `id`, which differs from its hash. The commit read and the repo read return no `id`, so import looks the commit up in the commit list, `GET /api/v1/commits/{owner}/{repo}`, whose entries carry both the `id` and the hash. -An import that stops after it creates the repo and before it pushes the commit leaves a prompt with no commit. The next import finds the prompt by its handle and, through the commit list, finds no commit. It then pushes the file's manifest as the first commit and creates the file's tags, as it does for a new prompt, without `--update`. The repo details it created hold the file's values already, and any that someone changed in between still need `--update`. +An import that stops after it creates the repo and before it pushes the commit leaves a prompt with no commit. The next import finds the prompt by its handle and, through the commit list, finds no commit. It then pushes the file's manifest as the first commit and creates the file's tags, as it does for a new prompt, without `--update`. The repo details it created hold the file's values already, and any that someone changed in between still need `--update`: without it, import pushes the commit and tags, leaves those repo details alone, lists them as differences and exits with status 3. With `--update`, import makes an existing assistant match the file and asks the Agent Server to remove what the file lacks. The Agent Server keeps the name, graph ID and description an update leaves out, and merges the metadata an update sends into the metadata the assistant already has, as `AssistantPatch` in `reference/openapi/langchain/agent-server/openapi.json` documents. So import sends the file's name, graph ID, configuration and context in full, an empty object for a configuration or context the file lacks, and a null value for each metadata key the destination has and the file lacks. `AssistantPatch` documents no way to clear a description, so in [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8) a developer sends a null and an empty description to the live server, along with a null metadata value and a replaced configuration and context, and records what the server keeps. Import then reads the assistant back, and any key or field the file lacks that is still there shows up as a difference, with exit status 3. The Agent Server saves each update as a new version of the assistant. diff --git a/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md index cfc63dfcf..b25fd1e3d 100644 --- a/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md +++ b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md @@ -187,7 +187,7 @@ When the destination workspace has no prompt with the file's handle, `langstar p ### Requirement: Finish an import that stopped before its first commit -When the destination workspace has a prompt with the file's handle and no commit, `langstar prompt import` SHALL push the file's manifest and commit description as the first commit and create the file's commit tags, as for a prompt that does not exist yet, without `--update`. Repo details that differ from the file SHALL still need `--update`. +When the destination workspace has a prompt with the file's handle and no commit, `langstar prompt import` SHALL push the file's manifest and commit description as the first commit and create the file's commit tags, as for a prompt that does not exist yet, without `--update`. Repo details that differ from the file SHALL still need `--update`: without it, import SHALL push the commit and tags, leave the repo details unchanged, list them as differences and exit with status 3. #### Scenario: Earlier import created the repo and stopped @@ -437,3 +437,13 @@ When `prompt compare`, `prompt verify` or `prompt import` prints either side of - **WHEN** a user runs `langstar prompt compare my-prompt.json --wrkspace-id ` - **THEN** langstar prints clap's usage error and exits with status 2 + +### Requirement: Report the import's action and destination ID in the JSON report + +With `-f json`, the object that `prompt import` and `assistant import` print SHALL also hold `action`, one of `create`, `update` and `unchanged`, and the destination's commit hash or assistant ID. With `--dry-run`, `action` SHALL name the action import would take, and the ID SHALL be null when import would create the resource. + +#### Scenario: Machine-readable first import + +- **WHEN** a user runs `langstar -f json prompt import my-prompt.json --workspace-id ` +- **AND** the destination has no prompt named `my-prompt` +- **THEN** stdout holds one JSON object whose `action` is `create` and that holds the new commit hash diff --git a/openspec/changes/export-import-prompts-and-assistants/tasks.md b/openspec/changes/export-import-prompts-and-assistants/tasks.md index 37cd31ce8..57c35e3c9 100644 --- a/openspec/changes/export-import-prompts-and-assistants/tasks.md +++ b/openspec/changes/export-import-prompts-and-assistants/tasks.md @@ -5,7 +5,7 @@ Each numbered group below is one pull request, merged in the order listed, excep ## 1. Check the API round trip before building on it - [ ] 1.1 Rerun the jq query for `prompt-endpoints.json` in `reference/api-specs/langsmith/FRAGMENTS.md`, so the extract picks up the `include` parameter of the commit read that its 2025-11-29 copy lacks, and extract the workspace secrets endpoints (`/api/v1/workspaces/current/secrets`) from `reference/openapi/langchain/langsmith/openapi.json` into a new file in `reference/api-specs/langsmith/` with its jq query in `FRAGMENTS.md`; verify by rerunning each query and getting the same file -- [ ] 1.2 Write `reference/research/807-openapi-validation.md`, recording what the live API returns. Verify that every claim in the report names the request that produced it. +- [ ] 1.2 Write `reference/research/807-openapi-validation.md`, recording what the live API returns. Verify that every claim in the report names the request that produced it. This task pushes to the second workspace and runs a prompt with a model, so it waits on the destination workspace and the OpenAI key that codekiln creates for task 2.1. - For a structured prompt saved with a model, read with `include=model`: the manifest, its outer `id`, the `model_config` and `model_provider` beside it, and whether `model_provider` matches the class name export writes to `requires.model_class`. - Push that manifest to a second workspace: whether the read-back returns the same manifest, `model_config` and `model_provider`, and what pushing an identical manifest again returns. - On the Agent Server: what `POST /assistants/search` returns for a graph ID the deployment does not serve, what it returns for `context`, whether an assistant update that sends a metadata key with a null value removes that key, whether an update that sends `config` or `context` replaces the assistant's object or merges into it, and whether an update can clear the description. @@ -92,6 +92,8 @@ Each integration test in this group carries `#[cfg_attr(not(feature = "integrati - An integration test that imports a file whose `requires.secrets` names a secret the destination workspace lacks, and asserts that it creates no prompt, names the secret and exits with status 3. - An integration test that creates a repo with the file's handle and repo details and no commit in the destination workspace, as an import that stopped after creating the repo leaves it, then imports the file without `--update`, and asserts that the prompt holds one commit with the file's manifest and the file's tags, and that import exits with status 0. - A CLI test that runs import without `--workspace-id` while `LANGSMITH_WORKSPACE_ID` is set, and asserts that it exits with status 1, names the missing flag and creates no prompt. + - A CLI test that `langstar -f json prompt import` of a new prompt prints one JSON object whose `action` is `create` and that holds the new commit hash, and that the same command with `--dry-run` prints `create` and a null commit hash. + - An integration test that imports a file whose repo readme differs from the destination's, into a destination that holds the repo and no commit, without `--update`, and asserts that import pushes the commit and tags, leaves the readme unchanged, reports `/repo/readme` and exits with status 3. - A CLI test that runs `prompt import` on an assistant export, and asserts that it exits with status 1, names `kind` and the value `assistant`, and creates no prompt. - CLI tests that import a file whose manifest refers to `ANTHROPIC_API_KEY` while `requires.secrets` holds only `OPENAI_API_KEY`, and a file whose `requires.model_class` names a class other than the one at the end of the manifest's chat model `id`, and assert that each creates no prompt, names the field and exits with status 1. - [ ] 7.2 Add `--update`, `--dry-run`, `--record` and the destination record. Verify with these tests: @@ -107,7 +109,7 @@ Each integration test in this group carries `#[cfg_attr(not(feature = "integrati - An integration test that `--update` against a destination whose latest commit matches the file and carries a `staging` tag the file lacks deletes `staging`, adds no commit and exits with status 0, and that `--dry-run` lists the delete. - An integration test that `--dry-run` writes nothing. - A unit test in `cli/src/commands/prompt.rs` that serves, through a `mockito` server and `LangchainClient::with_base_urls`, a commit read back that lacks a field the file holds, calls the import code, and asserts that the result names that field's JSON Pointer path and gives exit status 3, and that the destination record holds the new commit hash. -- [ ] 7.3 Add the structured output check as a mise task that runs a Python script with `uv run --script`. The script names `langsmith` and `langchain-openai` as inline script dependencies. It builds its `Client` with `workspace_id` set to `LANGSTAR_TEST_DEST_WORKSPACE_ID`, because both `fnox.toml` and CI set `LANGSMITH_WORKSPACE_ID` to the source workspace, which holds a prompt with the same handle. It then pulls the imported prompt with `include_model=True` and a `secrets` map holding the model provider key read from the environment, invokes it and validates the result against the schema. Add `uv` to `mise.toml`, and remove `install: false` from the Setup mise step of the Integration Tests job in `.github/workflows/ci.yml` so that job installs `uv` and Python 3.11. Verify by having the import integration test create a `json_schema` prompt and a `function_calling` prompt, each saved with a small OpenAI model, import both into the destination workspace and run the check on each, under `mise run check` and in that CI job. +- [ ] 7.3 Add the structured output check as a mise task that runs a Python script with `uv run --script`. The script names `langsmith`, `langchain-openai` and `jsonschema` as inline script dependencies, each pinned to an exact version, so a new upstream release cannot change the CI result, and validates each result with `jsonschema` against the prompt's schema. It builds its `Client` with `workspace_id` set to `LANGSTAR_TEST_DEST_WORKSPACE_ID`, because both `fnox.toml` and CI set `LANGSMITH_WORKSPACE_ID` to the source workspace, which holds a prompt with the same handle. It then pulls the imported prompt with `include_model=True` and a `secrets` map holding the model provider key read from the environment, invokes it and validates the result against the schema. Add `uv` to `mise.toml`, and remove `install: false` from the Setup mise step of the Integration Tests job in `.github/workflows/ci.yml` so that job installs `uv` and Python 3.11. Verify by having the import integration test create a `json_schema` prompt and a `function_calling` prompt, each saved with a small OpenAI model, import both into the destination workspace and run the check on each, under `mise run check` and in that CI job. - [ ] 7.4 Document prompt import, the workspace scope of prompts and setting up destination credentials; verify that each documented command runs as written ## 8. assistant export @@ -135,11 +137,11 @@ Each integration test in this group carries `#[cfg_attr(not(feature = "integrati - An integration test that runs verify with `--assistant-id` against an exported assistant and exits with status 0. - An integration test that exports an assistant, changes its metadata through the SDK, runs verify with `--assistant-id` set to that assistant, and asserts the reported path and exit status 3. - An integration test of a file whose graph ID the deployment lacks, which reports the graph ID and exits with status 3. + - Unit tests in `cli/src/commands/assistant.rs` that serve, through a `mockito` server, a search that answers 422 with an error naming `graph_id`, and assert that verify reports the graph ID missing and exits with status 3, and a 422 naming another field, and assert that verify exits with status 1. - An integration test of a file whose prompt tag the workspace lacks, which reports the prompt and exits with status 3. + - A unit test that a prompt reference with no owner and no tag, such as `support-system`, makes the prompt check request `GET /api/v1/commits/-/support-system/latest`, and one that `my-org/support-system:prod` requests `GET /api/v1/commits/my-org/support-system/prod`. - An integration test that runs verify with `--workspace-id` set to the destination workspace, for an assistant whose referenced prompt tag exists only in that workspace, and asserts that verify exits with status 0 and that `checks` reports the prompt as found. - - Unit tests in `cli/src/commands/assistant.rs` that serve, through a `mockito` server, a search that answers 422 with an error naming `graph_id`, and assert that verify reports the graph ID missing and exits with status 3, and a 422 naming another field, and assert that verify exits with status 1. - A CLI test that `langstar -f json assistant verify` prints one JSON object with `status`, `differences` and `checks` to stdout. - - A unit test that a prompt reference with no owner and no tag, such as `support-system`, makes the prompt check request `GET /api/v1/commits/-/support-system/latest`, and one that `my-org/support-system:prod` requests `GET /api/v1/commits/my-org/support-system/prod`. - [ ] 9.3 Document assistant compare and verify; verify that each documented command runs as written ## 10. assistant import From ae09fddfa6788b252563cb4c0688d03a0d7d2d9d Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Tue, 27 Oct 2026 17:43:34 -0400 Subject: [PATCH 76/90] =?UTF-8?q?=F0=9F=93=9D=20docs(openspec):=20decide?= =?UTF-8?q?=20six=20#807=20design=20questions=20and=20leave=20one=20for=20?= =?UTF-8?q?codekiln?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Compare, verify and import exit with 3 on a difference; import records destination IDs beside the export; --prompt-ref names prompt references; prompt import requires --workspace-id; a missing saved model configuration is a note; a failed dependency check stops import first. The structured output check skips without an OpenAI key, and the tests import into the existing workspace and deployment until codekiln answers the one remaining question. Co-Authored-By: Claude Opus 5.5 --- .../design.md | 90 ++++++------------- .../proposal.md | 2 +- .../tasks.md | 4 +- 3 files changed, 28 insertions(+), 68 deletions(-) diff --git a/openspec/changes/export-import-prompts-and-assistants/design.md b/openspec/changes/export-import-prompts-and-assistants/design.md index fb5543e2d..6df39082d 100644 --- a/openspec/changes/export-import-prompts-and-assistants/design.md +++ b/openspec/changes/export-import-prompts-and-assistants/design.md @@ -74,9 +74,9 @@ A credential under an ordinary key name with an ordinary-looking value, such as Export reads the commit with `include=model`, the form the Go SDK uses (`references/langsmith-go.md`). The Python SDK sends the older `include_model=true`, and the LangSmith OpenAPI spec accepts both. With the model included, the manifest wraps the prompt and the chat model with its settings in one constructor, and the response also carries `model_config` and `model_provider` beside the manifest (`references/langsmith-sdk.md`). No field of the create-commit request holds the model, so the model reaches the destination inside the manifest, and export keeps `model_config` and `model_provider` in the file only for the reader. The LangSmith Python SDK's [`PromptCommit`](https://github.com/langchain-ai/langsmith-sdk/blob/47c553caf08c1989e2be3ff1886da3eb801f709f/python/langsmith/schemas.py#L1054) documents `model_provider` as the model's class name, such as `ChatOpenAI`. In [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8), a developer pushes that manifest to a second workspace and records whether LangSmith returns the same manifest, `model_config` and `model_provider`, and which outer `id` the live API returns. -### Import records what it created in a separate file +### Import records the IDs it created in a separate file next to the export -Import writes the IDs of what it created to `.destinations.json`, keyed by destination workspace ID for prompts and by deployment ID for assistants. The export file stays the same after any number of imports, and a fresh export never wipes out the record of earlier imports. Import replaces the record by writing the whole new record to a temporary file in the same directory and renaming it over the old one, so a langstar that stops in the middle of a write leaves the earlier record whole, pending assistant IDs included. Langstar supports one import of an export file at a time, because a person runs import by hand from a Git checkout. When two imports of the same file run at once, each reads the record before the other writes it, and the import that writes last drops the other's entries. Two first imports of one assistant into the same deployment can also create two assistants. [#828 make concurrent prompt and assistant imports into one deployment safe](https://github.com/codekiln/langstar/issues/828) covers making concurrent imports safe. +Keeping the record out of the export file means export never has to carry an earlier record forward when it overwrites the file, and the file a reviewer reads in Git changes only when the source changes. Import writes the IDs of what it created to `.destinations.json`, keyed by destination workspace ID for prompts and by deployment ID for assistants. The export file stays the same after any number of imports, and a fresh export never wipes out the record of earlier imports. Import replaces the record by writing the whole new record to a temporary file in the same directory and renaming it over the old one, so a langstar that stops in the middle of a write leaves the earlier record whole, pending assistant IDs included. Langstar supports one import of an export file at a time, because a person runs import by hand from a Git checkout. When two imports of the same file run at once, each reads the record before the other writes it, and the import that writes last drops the other's entries. Two first imports of one assistant into the same deployment can also create two assistants. [#828 make concurrent prompt and assistant imports into one deployment safe](https://github.com/codekiln/langstar/issues/828) covers making concurrent imports safe. A repeated prompt import finds the copy by its handle and uses the record to tell whether someone changed it. Import writes the new commit hash to the record before it reads the commit back, so a read-back that finds a difference still leaves the hash in the record. The next import compares the destination's latest commit with that hash, and reports a change only when someone else pushed a commit after langstar's. Langstar can stop after it pushes a commit and before it writes that commit's hash to the record. The next import finds the destination's latest commit matching the file, writes its hash to the record and reports the prompt unchanged. @@ -102,27 +102,27 @@ Compare reads the prompt or assistant from the destination, puts it in the shape Prompt verify checks that the workspace holds every secret the prompt's model reads. Export lists those names in `requires.secrets` by walking the manifest for LangChain secret references, `{"lc": 1, "type": "secret", "id": ["OPENAI_API_KEY"]}`, the shape the Python SDK's `_manifest_has_secrets` matches. Verify looks each name up among the workspace's secrets (`GET /api/v1/workspaces/current/secrets`, which `langstar secrets list` already calls) and prints the name of each missing secret and no secret value. -Prompt verify also notes when the workspace has no saved model configuration for the prompt's model, and leaves the exit status alone, because the prompt runs with the model its manifest carries. Export takes the model's class name, such as `ChatOpenAI`, from the last element of the chat model constructor's `id` in the manifest, such as `["langchain", "chat_models", "openai", "ChatOpenAI"]`, and stores it in `requires.model_class`. Verify looks for a saved model configuration (`GET /api/v1/playground-settings`, which `langstar model-config list` calls) whose settings `id` ends in that class name. A prompt saved without a model has no chat model constructor in its manifest, so export writes `requires.model_class` as null, and verify and import skip this check for it and still run the secret check. [#807 Export, import, compare, and verify prompts and assistants across LangSmith environments](https://github.com/codekiln/langstar/issues/807) asks for the note, and [Should prompt verify report a missing saved model configuration?](#7---should-prompt-verify-report-a-missing-saved-model-configuration) asks whether to keep it. +Prompt verify also notes when the workspace has no saved model configuration for the prompt's model, and leaves the exit status alone, because the prompt runs with the model its manifest carries. Export takes the model's class name, such as `ChatOpenAI`, from the last element of the chat model constructor's `id` in the manifest, such as `["langchain", "chat_models", "openai", "ChatOpenAI"]`, and stores it in `requires.model_class`. Verify looks for a saved model configuration (`GET /api/v1/playground-settings`, which `langstar model-config list` calls) whose settings `id` ends in that class name. A prompt saved without a model has no chat model constructor in its manifest, so export writes `requires.model_class` as null, and verify and import skip this check for it and still run the secret check. [Prompt verify reports a missing saved model configuration as a note, not a failure](#prompt-verify-reports-a-missing-saved-model-configuration-as-a-note-not-a-failure) gives the reason. Import and verify work out `requires` again from the file's manifest: the secret names, and the model class or null. When either differs from what the file holds, they write nothing, name the field and exit with status 1. This catches a manifest someone edited in Git without updating `requires`. Assistant verify checks the graph with `POST /assistants/search` filtered to the graph ID; every graph on a deployment has a default assistant, so an empty result means the deployment does not serve the graph. The Agent Server's OpenAPI spec lists the deployment's graph IDs as the allowed values of the search's `graph_id`, so the server may answer an unknown graph ID with 422 instead of an empty result. Verify therefore also counts a 422 whose error names `graph_id` as a missing graph, and exits with status 1 on any other error; in [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8) a developer records which answer the live server gives. It checks each prompt in `requires.prompts`, such as `support-system:prod`, by asking LangSmith for the commit that tag points at (`GET /api/v1/commits/{owner}/{repo}/{tag}`, the request `langstar prompt pull --commit prod` already sends). A prompt name has the form `[/][:]`: without an owner verify uses `-`, the current workspace, and without a tag it asks for `latest`. A 404 means the workspace lacks the prompt or its tag. -### Import checks the destination's secrets, graph and prompts before it writes +### A failed dependency check stops import before it writes anything -Before it writes anything, import checks that the destination has the secrets, the graph and the prompts the copy needs. These are the checks `verify` makes, and import runs them because [#807 Export, import, compare, and verify prompts and assistants across LangSmith environments](https://github.com/codekiln/langstar/issues/807) asks import to check the destination's dependencies. Prompt import stops when the destination workspace lacks a secret in `requires.secrets`, and assistant import stops when the deployment does not serve the file's graph or the workspace, from `--workspace-id` or the configured workspace, lacks a prompt in `requires.prompts`. Each names what is missing, writes nothing and exits with status 3, and `--dry-run` reports the same failed checks. A user therefore creates the destination workspace's secrets before importing a prompt, and imports the prompts before importing an assistant that refers to them. [Should a failed dependency check stop import before it writes?](#8---should-a-failed-dependency-check-stop-import-before-it-writes) asks whether import should write anyway and report the failed checks afterwards. +A copy that cannot run is worse than no copy, because the user then has to find and delete it by hand; asking for the destination's secrets and prompts first costs only the order of the steps. Before it writes anything, import checks that the destination has the secrets, the graph and the prompts the copy needs. These are the checks `verify` makes, and import runs them because [#807 Export, import, compare, and verify prompts and assistants across LangSmith environments](https://github.com/codekiln/langstar/issues/807) asks import to check the destination's dependencies. Prompt import stops when the destination workspace lacks a secret in `requires.secrets`, and assistant import stops when the deployment does not serve the file's graph or the workspace, from `--workspace-id` or the configured workspace, lacks a prompt in `requires.prompts`. Each names what is missing, writes nothing and exits with status 3, and `--dry-run` reports the same failed checks. A user therefore creates the destination workspace's secrets before importing a prompt, and imports the prompts before importing an assistant that refers to them. -### The user names an assistant's prompt references +### Users name an assistant's prompt references with `--prompt-ref` -An assistant refers to a prompt through whatever configuration key its graph reads, so langstar cannot find the references by itself. `assistant export --prompt-ref ` names each one. The assistant's configuration should name each prompt by handle and tag, such as `support-system:prod`, because the same prompt gets a different commit hash in each workspace. Import copies the commit tags to the destination workspace, so when the graph on the destination deployment asks LangSmith for `support-system:prod`, it gets the same prompt. Compare, verify and import look up each pointer from `requires.prompts` in the file's `assistant` object, and exit with status 1 when the value there differs from the prompt name recorded beside the pointer. This catches a configuration someone edited in Git without updating `requires.prompts`. +A graph reads its prompt names from whatever configuration key its author chose, and guessing from strings that look like prompt handles would miss some and flag others, so the user says where they are. An assistant refers to a prompt through whatever configuration key its graph reads, so langstar cannot find the references by itself. `assistant export --prompt-ref ` names each one. The assistant's configuration should name each prompt by handle and tag, such as `support-system:prod`, because the same prompt gets a different commit hash in each workspace. Import copies the commit tags to the destination workspace, so when the graph on the destination deployment asks LangSmith for `support-system:prod`, it gets the same prompt. Compare, verify and import look up each pointer from `requires.prompts` in the file's `assistant` object, and exit with status 1 when the value there differs from the prompt name recorded beside the pointer. This catches a configuration someone edited in Git without updating `requires.prompts`. ### compare, verify and import exit with 3 when they find a difference -Status 0 means the resource matches and every check passed. Status 3 means langstar found a difference or a failed check. Status 1 means any other error. Status 2 stays clap's: `Cli::parse()` in `run` in `cli/src/main.rs` exits with 2 when a user mistypes a flag or leaves out a required one, and this change leaves that alone, so a script never reads a typo as a difference. `prompt import` declares `--workspace-id` optional, as the other `langstar prompt` subcommands do, and checks for the flag itself. When a user leaves the flag out, langstar exits with status 1 and an error that names `--workspace-id`, where clap would have exited with 2. `diff` uses 1 for differences and 2 for errors; langstar keeps 1 for errors so that a script testing for failure sees the same status from every command. +Langstar exits today only with 0, with 1 from `main` in `cli/src/main.rs` when a command returns an error, and with clap's 2, so 3 is the first status that means neither success nor a broken run. Status 0 means the resource matches and every check passed. Status 3 means langstar found a difference or a failed check. Status 1 means any other error. Status 2 stays clap's: `Cli::parse()` in `run` in `cli/src/main.rs` exits with 2 when a user mistypes a flag or leaves out a required one, and this change leaves that alone, so a script never reads a typo as a difference. `prompt import` declares `--workspace-id` optional, as the other `langstar prompt` subcommands do, and checks for the flag itself. When a user leaves the flag out, langstar exits with status 1 and an error that names `--workspace-id`, where clap would have exited with 2. `diff` uses 1 for differences and 2 for errors; langstar keeps 1 for errors so that a script testing for failure sees the same status from every command. -### A Python script checks that an imported structured prompt still returns structured data +### An optional Python script checks that an imported structured prompt still returns structured data -The LangSmith API has no documented way to run a prompt with its model, so a short Python script runs each imported structured prompt with LangChain and checks that the result matches the prompt's schema. The one prompt endpoint in the OpenAPI spec, `POST /api/v1/prompts/invoke_prompt`, takes message templates and inputs with no model or schema, and no LangSmith SDK calls it; in [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8), a developer calls it on the live API in case it does more than the spec says. A mise task runs the script with `uv run --script`, which installs the `langsmith` and `langchain-openai` packages the script lists, on a developer's machine and in the Integration Tests CI job. The script reads the destination workspace: it builds its LangSmith `Client` with `workspace_id` set to `LANGSTAR_TEST_DEST_WORKSPACE_ID`, because both `fnox.toml` and CI set `LANGSMITH_WORKSPACE_ID` to the source workspace, which holds a prompt with the same handle. The script pulls the imported prompt with `include_model=True` and passes `secrets={"OPENAI_API_KEY": }` to [`pull_prompt`](https://github.com/langchain-ai/langsmith-sdk/blob/47c553caf08c1989e2be3ff1886da3eb801f709f/python/langsmith/client.py#L10228), because the LangSmith Python SDK reads no secret from the environment unless asked, and raises when a manifest holds a secret reference and no `secrets` map arrives. The script needs an OpenAI API key. codekiln creates it and stores it in 1Password, in the repository's CI secrets and as the destination workspace's `OPENAI_API_KEY` secret, and in [export-import-prompts-and-assistants Task 2.1 Add the destination workspace and the model provider key](tasks.md#L16) a developer adds it to `fnox.toml` and the CI workflow. +The LangSmith API has no documented way to run a prompt with its model, so a short Python script runs each imported structured prompt with LangChain and checks that the result matches the prompt's schema. The one prompt endpoint in the OpenAPI spec, `POST /api/v1/prompts/invoke_prompt`, takes message templates and inputs with no model or schema, and no LangSmith SDK calls it; in [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8), a developer calls it on the live API in case it does more than the spec says. A mise task runs the script with `uv run --script`, which installs the `langsmith` and `langchain-openai` packages the script lists, on a developer's machine and in the Integration Tests CI job. The script reads the destination workspace: it builds its LangSmith `Client` with `workspace_id` set to `LANGSTAR_TEST_DEST_WORKSPACE_ID`, because both `fnox.toml` and CI set `LANGSMITH_WORKSPACE_ID` to the source workspace, which holds a prompt with the same handle. The script pulls the imported prompt with `include_model=True` and passes `secrets={"OPENAI_API_KEY": }` to [`pull_prompt`](https://github.com/langchain-ai/langsmith-sdk/blob/47c553caf08c1989e2be3ff1886da3eb801f709f/python/langsmith/client.py#L10228), because the LangSmith Python SDK reads no secret from the environment unless asked, and raises when a manifest holds a secret reference and no `secrets` map arrives. The script needs an OpenAI API key, which bills on every run. The check is optional: the mise task skips it and says so when `OPENAI_API_KEY` is unset, so every other test runs without a paid key, and a developer with a key of their own can run it by hand. The question under Open Questions asks for a key in CI. ### Export and import use the same SDK calls as the planned `prompt create`, `get` and `update` @@ -140,6 +140,18 @@ Developers commit the SDK's new fields and calls as `✨ feat(sdk)`, with no `BR Developers leave the structs as they are, without `#[non_exhaustive]`, because adding that attribute would stop the CLI and the SDK's integration tests from building `Assistant`, `CommitRequest`, `CreateAssistantRequest` and `UpdateAssistantRequest` with struct literals, which [docs/dev/procedures.md:116](https://github.com/codekiln/langstar/blob/ab158b14160cf7803a79384f88756911bcd21857/docs/dev/procedures.md#L116) counts as a breaking change. +### Prompt import requires `--workspace-id`, even when `LANGSMITH_WORKSPACE_ID` is set + +Import writes, and `LANGSMITH_WORKSPACE_ID` is easy to leave pointing at the wrong workspace: `fnox.toml` and CI set it to the source workspace, so a fallback would put the copy back where it came from without a word. Requiring the flag puts the destination on the command line, where the user and the shell history can see it, as assistant import does with `--deployment`. + +### Prompt verify reports a missing saved model configuration as a note, not a failure + +The prompt's manifest carries the model and its settings, so the prompt runs without a saved model configuration, and the secret check is the one that finds a prompt that cannot authenticate. Failing on the note would make verify exit with 3 for a prompt that runs. + +### Without a second workspace and deployment, the tests import into the existing ones + +Until a second workspace and a second test deployment exist, the integration tests import each prompt into the existing test workspace under a fresh handle, and each assistant onto the existing test deployment as a new assistant. That covers import, compare, verify and the destination record, but not a move between workspaces or deployments, and the test that moves an assistant to `test_graph_alt` is skipped. The question under Open Questions asks for both. + ## Risks / Trade-offs - [LangSmith may store a pushed model differently from the source] → Import reads the prompt back and reports any difference, and in [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8) a developer checks the round trip against the live API before anyone builds the commands. @@ -150,7 +162,7 @@ Developers leave the structs as they are, without `#[non_exhaustive]`, because a - [The check flags a string that holds no secret, such as an example token in a message template] → Export names the path and the rule, and the user passes `--not-a-secret` with that path after reading the string. - [A tag in the destination may already point at another commit] → `--update` moves the tag, and `--dry-run` lists the move. - [`--update` deletes a tag on the matching destination commit that the file lacks, and a graph on a deployment may read that tag] → `--dry-run` lists each delete, and `assistant verify` reports a prompt tag an assistant refers to that the workspace no longer has. -- [The control plane may refuse a second deployment from this repository with 409] → In [export-import-prompts-and-assistants Task 2.3 Register the test graph under a second graph ID](tasks.md#L18), a developer deploys the new `langgraph.json` once, before anyone builds the fixture. If the control plane answers with 409, the developer brings [Which second workspace and second deployment should the tests use?](#5---which-second-workspace-and-second-deployment-should-the-tests-use) back to codekiln. +- [The control plane may refuse a second deployment from this repository with 409] → In [export-import-prompts-and-assistants Task 2.3 Register the test graph under a second graph ID](tasks.md#L18), a developer deploys the new `langgraph.json` once, before anyone builds the fixture. If the control plane answers with 409, the developer reports it to codekiln with the question under Open Questions. ## Resolved Questions @@ -158,58 +170,6 @@ None. ## Open Questions -### 1 - Should compare, verify and import exit with 3 when they find a difference? - -> Context from the drafting agent, for question 1. -> Status 2 is taken: clap exits with 2 when a user mistypes a flag or leaves out a required one, so a difference reported as 2 would look like a typo to a script. The draft uses 0 for a match, 3 for a difference or failed check and 1 for every other error, and leaves clap's 2 alone. The other choice makes `run` in `cli/src/main.rs` exit with 1 on a usage error, which frees 2 for differences but changes the status every langstar command gives for a mistyped flag. `diff` and `git diff --exit-code` use 1 for differences, which would make a difference look like an error. Recommendation: 3 for a difference. - - - -### 2 - Should import record destination IDs in a separate file next to the export? - -> Context from the drafting agent, for question 2. -> The draft writes `.destinations.json` and leaves the export unchanged. Keeping the record inside the export file would make one file per resource, but export would then have to keep the record when it overwrites the file. Recommendation: the separate file. - - - -### 3 - How should the end-to-end test show that an imported structured prompt still returns structured data? - -> Context from the drafting agent, for question 3. -> The LangSmith OpenAPI spec's only prompt-invoke endpoint, `POST /api/v1/prompts/invoke_prompt`, takes message templates and inputs with no model or schema, and no SDK calls it. In [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8), a developer calls it on the live API. The draft runs a Python script through a mise task with `uv run --script`. The script pulls the prompt from the destination workspace with the LangSmith Python SDK's `pull_prompt`, runs it with LangChain, and names both packages as inline dependencies. [export-import-prompts-and-assistants Task 7.3 Add the structured output check](tasks.md#L110) adds `uv` to `mise.toml` and removes `install: false` from the Integration Tests CI job, so that job runs `mise install`. It needs a model provider key, such as `OPENAI_API_KEY`, which codekiln would create and store in 1Password, the CI secrets and the destination workspace's secrets. Building the model call into langstar as `prompt verify --invoke` would need langstar to call each provider's API itself. Recommendation: the Python script, with an OpenAI key and a small model. - - - -### 4 - Should the user name an assistant's prompt references with --prompt-ref? - -> Context from the drafting agent, for question 4. -> A graph reads its prompt names from whatever configuration key its author chose, so langstar cannot find them reliably. Guessing from strings that look like prompt handles would miss some and flag others. Recommendation: `--prompt-ref `, repeatable, recorded in the export file so import and verify reuse it. - - - -### 5 - Which second workspace and second deployment should the tests use? - -> Context from the drafting agent, for question 5. -> The import tests need a workspace and a deployment to import into. The draft has codekiln create a second workspace in the test organization, which the existing API key can reach, and the tests read its ID from `LANGSTAR_TEST_DEST_WORKSPACE_ID`. The destination deployment fixture from [export-import-prompts-and-assistants Task 2.4 Add a destination deployment fixture](tasks.md#L19) deploys the test graph a second time from a `langgraph.json` of its own, which registers the graph as both `test_graph` and `test_graph_alt` so the update tests can move an assistant from one graph to the other. Nobody knows yet whether the control plane accepts that deployment. It answers a second deployment of the same agent environment with 409 "A deployment already exists for this agent environment", and no document says whether a second `langgraph.json` in the same repository makes a new agent environment. In [export-import-prompts-and-assistants Task 2.3 Register the test graph under a second graph ID](tasks.md#L18), a developer deploys the new `langgraph.json` once to find out. If the control plane answers with the 409, the developer stops before task 2.4 and brings this question back to codekiln with what the control plane returned. `.github/workflows/cleanup-test-deployments.yml` deletes each test deployment once it is four hours old and the fixture rebuilds it on the next CI run, so the first CI run after each cleanup waits for two deployments to build instead of one. Recommendation: a second workspace in the same organization, which codekiln creates, and one destination deployment that the fixture from task 2.4 rebuilds whenever the cleanup workflow has deleted it. - - - -### 6 - Should prompt import refuse to run without --workspace-id even when LANGSMITH_WORKSPACE_ID is set? - -> Context from the drafting agent, for question 6. -> [#807 Export, import, compare, and verify prompts and assistants across LangSmith environments](https://github.com/codekiln/langstar/issues/807) asks for an explicit destination workspace. Requiring the flag means langstar imports into the workspace the user typed on the command line, even when `LANGSMITH_WORKSPACE_ID` names another, at the cost of one more flag in scripts. Assistant import already requires `--deployment`. Recommendation: require the flag. - - - -### 7 - Should prompt verify report a missing saved model configuration? - -> Context from the drafting agent, for question 7. -> [#807 Export, import, compare, and verify prompts and assistants across LangSmith environments](https://github.com/codekiln/langstar/issues/807) asks import to report missing model configurations, so the draft has prompt verify look for a saved model configuration (`GET /api/v1/playground-settings`) whose settings `id` ends in the class name the export file holds in `requires.model_class`, such as `ChatOpenAI`. Nothing fails at run time without one, because the prompt's manifest carries the model and its settings; the secret check is the one that finds a prompt that cannot authenticate. Reporting it as a failure would make verify exit with 3 for a prompt that runs. Recommendation: report a missing saved model configuration as a note that leaves the exit status alone. - - - -### 8 - Should a failed dependency check stop import before it writes? - -> Context from the drafting agent, for question 8. -> [#807 Export, import, compare, and verify prompts and assistants across LangSmith environments](https://github.com/codekiln/langstar/issues/807) asks import to check the destination's dependencies: the secrets a prompt's model reads, and an assistant's graph and referenced prompts. The draft runs those checks before import writes and writes nothing when one fails, so a user never has to find and delete a copy that cannot run. The cost is order: the destination workspace needs its secrets before a prompt import, and its prompts before an assistant import. The other choice writes the copy anyway and reports the failed checks with status 3 afterwards, which lets a user import in any order and fix the destination later. Recommendation: stop before writing. +### q01 - A second workspace, a second test deployment and a paid OpenAI key - +Without them, the tests import into the existing test workspace and deployment, never cover a move between environments, and skip the structured output check. With them, the tests copy across environments and CI runs each imported structured prompt once, billed per run. May I create a second LangSmith workspace and a second test deployment, and use a paid OpenAI key in CI? diff --git a/openspec/changes/export-import-prompts-and-assistants/proposal.md b/openspec/changes/export-import-prompts-and-assistants/proposal.md index d16f04ad9..5c7ec6f60 100644 --- a/openspec/changes/export-import-prompts-and-assistants/proposal.md +++ b/openspec/changes/export-import-prompts-and-assistants/proposal.md @@ -33,7 +33,7 @@ A prompt belongs to a workspace and an assistant belongs to a deployment, but bo - SDK: `sdk/src/prompts.rs` gains the prompt calls export and import need: a commit read that asks for the model and returns it on `CommitManifestResponse`, a commit create that sends a description and returns the new commit's `id`, a commit list, calls that list, create, move and delete commit tags, and a repo update. `Prompt` gains the repo's readme, tags and `tenant_id`, and export fills `source.workspace_id` from `tenant_id`. `sdk/src/assistants.rs` gains the `context` and `version` fields, `assistant_id` and `if_exists` on create, `graph_id` on update, and a search by graph ID. Developers commit these fields as `✨ feat(sdk)` with no `BREAKING CHANGE:` footer, so the Prepare Release workflow makes the next release a minor version. [design.md's decision on the SDK release](design.md#developers-commit-the-new-sdk-fields-as--featsdk-so-the-next-release-is-a-minor-version) links the release docs that ask for this. - CLI: new subcommands in `cli/src/commands/prompt.rs` and `cli/src/commands/assistant.rs`, and a shared module for the export file, the destination record, comparison and the exit status. -- Tests need a second workspace, a second deployment and one model provider key for the structured output check. codekiln creates the workspace and the key, and langstar's tests read them through `fnox.toml` and the CI workflow; the destination deployment fixture from [export-import-prompts-and-assistants Task 2.4 Add a destination deployment fixture](tasks.md#L19) creates the deployment whenever the cleanup workflow has deleted it, as the source fixture does today. Nobody knows yet whether the control plane accepts a second deployment from this repository. In [export-import-prompts-and-assistants Task 2.3 Register the test graph under a second graph ID](tasks.md#L18), a developer deploys the new `langgraph.json` once to find out, before anyone builds the fixture. +- Tests that copy between environments need a second workspace, a second deployment and one model provider key for the structured output check, which the question under Open Questions in `design.md` asks codekiln for. Until then, the tests import into the existing test workspace and deployment and the structured output check skips. Once they exist, langstar's tests read them through `fnox.toml` and the CI workflow; the destination deployment fixture from [export-import-prompts-and-assistants Task 2.4 Add a destination deployment fixture](tasks.md#L19) creates the deployment whenever the cleanup workflow has deleted it, as the source fixture does today. Nobody knows yet whether the control plane accepts a second deployment from this repository. In [export-import-prompts-and-assistants Task 2.3 Register the test graph under a second graph ID](tasks.md#L18), a developer deploys the new `langgraph.json` once to find out, before anyone builds the fixture. - Docs: a new usage page on moving prompts and assistants between environments. ## Citations diff --git a/openspec/changes/export-import-prompts-and-assistants/tasks.md b/openspec/changes/export-import-prompts-and-assistants/tasks.md index 57c35e3c9..522507405 100644 --- a/openspec/changes/export-import-prompts-and-assistants/tasks.md +++ b/openspec/changes/export-import-prompts-and-assistants/tasks.md @@ -1,6 +1,6 @@ # Tasks -Each numbered group below is one pull request, merged in the order listed, except group 2, whose tasks each get a pull request of their own. Tasks 2.3 and 2.4 merge in that order: the fixture in 2.4 deploys from `main`, so the `langgraph.json` from 2.3 has to be on `main` first. Task 2.2 can merge at any time. Task 2.1 waits on codekiln to create the destination workspace and an OpenAI key, and can merge any time before group 6, whose prompt compare and verify tests are the first to use that workspace. +Each numbered group below is one pull request, merged in the order listed, except group 2, whose tasks each get a pull request of their own. Tasks 2.3 and 2.4 merge in that order: the fixture in 2.4 deploys from `main`, so the `langgraph.json` from 2.3 has to be on `main` first. Task 2.2 can merge at any time. Tasks 2.1, 2.3 and 2.4 wait on codekiln's answer to the question under Open Questions in `design.md`. Until then, the tests import into the existing test workspace and deployment, as `design.md`'s decision on test environments says, and the structured output check in 7.3 skips. ## 1. Check the API round trip before building on it @@ -109,7 +109,7 @@ Each integration test in this group carries `#[cfg_attr(not(feature = "integrati - An integration test that `--update` against a destination whose latest commit matches the file and carries a `staging` tag the file lacks deletes `staging`, adds no commit and exits with status 0, and that `--dry-run` lists the delete. - An integration test that `--dry-run` writes nothing. - A unit test in `cli/src/commands/prompt.rs` that serves, through a `mockito` server and `LangchainClient::with_base_urls`, a commit read back that lacks a field the file holds, calls the import code, and asserts that the result names that field's JSON Pointer path and gives exit status 3, and that the destination record holds the new commit hash. -- [ ] 7.3 Add the structured output check as a mise task that runs a Python script with `uv run --script`. The script names `langsmith`, `langchain-openai` and `jsonschema` as inline script dependencies, each pinned to an exact version, so a new upstream release cannot change the CI result, and validates each result with `jsonschema` against the prompt's schema. It builds its `Client` with `workspace_id` set to `LANGSTAR_TEST_DEST_WORKSPACE_ID`, because both `fnox.toml` and CI set `LANGSMITH_WORKSPACE_ID` to the source workspace, which holds a prompt with the same handle. It then pulls the imported prompt with `include_model=True` and a `secrets` map holding the model provider key read from the environment, invokes it and validates the result against the schema. Add `uv` to `mise.toml`, and remove `install: false` from the Setup mise step of the Integration Tests job in `.github/workflows/ci.yml` so that job installs `uv` and Python 3.11. Verify by having the import integration test create a `json_schema` prompt and a `function_calling` prompt, each saved with a small OpenAI model, import both into the destination workspace and run the check on each, under `mise run check` and in that CI job. +- [ ] 7.3 Add the structured output check as a mise task that runs a Python script with `uv run --script`. The script names `langsmith`, `langchain-openai` and `jsonschema` as inline script dependencies, each pinned to an exact version, so a new upstream release cannot change the CI result, and validates each result with `jsonschema` against the prompt's schema. It builds its `Client` with `workspace_id` set to `LANGSTAR_TEST_DEST_WORKSPACE_ID`, because both `fnox.toml` and CI set `LANGSMITH_WORKSPACE_ID` to the source workspace, which holds a prompt with the same handle. It then pulls the imported prompt with `include_model=True` and a `secrets` map holding the model provider key read from the environment, invokes it and validates the result against the schema. Add `uv` to `mise.toml`, and remove `install: false` from the Setup mise step of the Integration Tests job in `.github/workflows/ci.yml` so that job installs `uv` and Python 3.11. Verify by having the import integration test create a `json_schema` prompt and a `function_calling` prompt, each saved with a small OpenAI model, import both into the destination workspace and run the check on each, under `mise run check` and in that CI job. When `OPENAI_API_KEY` is unset, the task prints that it skipped the check and exits with status 0; verify that too, by running it without the key. - [ ] 7.4 Document prompt import, the workspace scope of prompts and setting up destination credentials; verify that each documented command runs as written ## 8. assistant export From a0140f65e7cb235494bb8442c818d7d15d39e7b9 Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Tue, 27 Oct 2026 17:44:48 -0400 Subject: [PATCH 77/90] =?UTF-8?q?=F0=9F=93=9A=20docs(openspec):=20run=20th?= =?UTF-8?q?e=20tests=20in=20the=20existing=20workspace=20and=20deployment?= =?UTF-8?q?=20and=20drop=20the=20last=20open=20question?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The structured output check skips without an OpenAI key, and the tests import into the single existing workspace and test deployment. Trying export and import on real assistants and prompts before implementation settles whether a second workspace and deployment are needed. Co-Authored-By: Claude Opus 5.5 --- .../export-import-prompts-and-assistants/design.md | 12 +++++------- .../export-import-prompts-and-assistants/proposal.md | 2 +- .../export-import-prompts-and-assistants/tasks.md | 6 +++--- 3 files changed, 9 insertions(+), 11 deletions(-) diff --git a/openspec/changes/export-import-prompts-and-assistants/design.md b/openspec/changes/export-import-prompts-and-assistants/design.md index 6df39082d..c0e55115a 100644 --- a/openspec/changes/export-import-prompts-and-assistants/design.md +++ b/openspec/changes/export-import-prompts-and-assistants/design.md @@ -122,7 +122,7 @@ Langstar exits today only with 0, with 1 from `main` in `cli/src/main.rs` when a ### An optional Python script checks that an imported structured prompt still returns structured data -The LangSmith API has no documented way to run a prompt with its model, so a short Python script runs each imported structured prompt with LangChain and checks that the result matches the prompt's schema. The one prompt endpoint in the OpenAPI spec, `POST /api/v1/prompts/invoke_prompt`, takes message templates and inputs with no model or schema, and no LangSmith SDK calls it; in [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8), a developer calls it on the live API in case it does more than the spec says. A mise task runs the script with `uv run --script`, which installs the `langsmith` and `langchain-openai` packages the script lists, on a developer's machine and in the Integration Tests CI job. The script reads the destination workspace: it builds its LangSmith `Client` with `workspace_id` set to `LANGSTAR_TEST_DEST_WORKSPACE_ID`, because both `fnox.toml` and CI set `LANGSMITH_WORKSPACE_ID` to the source workspace, which holds a prompt with the same handle. The script pulls the imported prompt with `include_model=True` and passes `secrets={"OPENAI_API_KEY": }` to [`pull_prompt`](https://github.com/langchain-ai/langsmith-sdk/blob/47c553caf08c1989e2be3ff1886da3eb801f709f/python/langsmith/client.py#L10228), because the LangSmith Python SDK reads no secret from the environment unless asked, and raises when a manifest holds a secret reference and no `secrets` map arrives. The script needs an OpenAI API key, which bills on every run. The check is optional: the mise task skips it and says so when `OPENAI_API_KEY` is unset, so every other test runs without a paid key, and a developer with a key of their own can run it by hand. The question under Open Questions asks for a key in CI. +The LangSmith API has no documented way to run a prompt with its model, so a short Python script runs each imported structured prompt with LangChain and checks that the result matches the prompt's schema. The one prompt endpoint in the OpenAPI spec, `POST /api/v1/prompts/invoke_prompt`, takes message templates and inputs with no model or schema, and no LangSmith SDK calls it; in [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8), a developer calls it on the live API in case it does more than the spec says. A mise task runs the script with `uv run --script`, which installs the `langsmith` and `langchain-openai` packages the script lists, on a developer's machine and in the Integration Tests CI job. The script reads the destination workspace: it builds its LangSmith `Client` with `workspace_id` set to `LANGSTAR_TEST_DEST_WORKSPACE_ID`, because both `fnox.toml` and CI set `LANGSMITH_WORKSPACE_ID` to the source workspace, which holds a prompt with the same handle. The script pulls the imported prompt with `include_model=True` and passes `secrets={"OPENAI_API_KEY": }` to [`pull_prompt`](https://github.com/langchain-ai/langsmith-sdk/blob/47c553caf08c1989e2be3ff1886da3eb801f709f/python/langsmith/client.py#L10228), because the LangSmith Python SDK reads no secret from the environment unless asked, and raises when a manifest holds a secret reference and no `secrets` map arrives. The script needs an OpenAI API key, which bills on every run. The check is optional: the mise task skips it and says so when `OPENAI_API_KEY` is unset, so every other test runs without a paid key, and a developer with a key of their own can run it by hand. ### Export and import use the same SDK calls as the planned `prompt create`, `get` and `update` @@ -148,9 +148,9 @@ Import writes, and `LANGSMITH_WORKSPACE_ID` is easy to leave pointing at the wro The prompt's manifest carries the model and its settings, so the prompt runs without a saved model configuration, and the secret check is the one that finds a prompt that cannot authenticate. Failing on the note would make verify exit with 3 for a prompt that runs. -### Without a second workspace and deployment, the tests import into the existing ones +### The tests run in the single existing workspace and test deployment -Until a second workspace and a second test deployment exist, the integration tests import each prompt into the existing test workspace under a fresh handle, and each assistant onto the existing test deployment as a new assistant. That covers import, compare, verify and the destination record, but not a move between workspaces or deployments, and the test that moves an assistant to `test_graph_alt` is skipped. The question under Open Questions asks for both. +The integration tests import each prompt into the existing test workspace under a fresh handle, and each assistant onto the existing test deployment as a new assistant. That covers import, compare, verify and the destination record, but not a move between workspaces or deployments, and the test that moves an assistant to `test_graph_alt` is skipped. Whether the tests need a second workspace and deployment will be settled by trying export and import on real assistants and prompts before implementation, and the result will be recorded here. ## Risks / Trade-offs @@ -162,7 +162,7 @@ Until a second workspace and a second test deployment exist, the integration tes - [The check flags a string that holds no secret, such as an example token in a message template] → Export names the path and the rule, and the user passes `--not-a-secret` with that path after reading the string. - [A tag in the destination may already point at another commit] → `--update` moves the tag, and `--dry-run` lists the move. - [`--update` deletes a tag on the matching destination commit that the file lacks, and a graph on a deployment may read that tag] → `--dry-run` lists each delete, and `assistant verify` reports a prompt tag an assistant refers to that the workspace no longer has. -- [The control plane may refuse a second deployment from this repository with 409] → In [export-import-prompts-and-assistants Task 2.3 Register the test graph under a second graph ID](tasks.md#L18), a developer deploys the new `langgraph.json` once, before anyone builds the fixture. If the control plane answers with 409, the developer reports it to codekiln with the question under Open Questions. +- [The control plane may refuse a second deployment from this repository with 409] → In [export-import-prompts-and-assistants Task 2.3 Register the test graph under a second graph ID](tasks.md#L18), a developer deploys the new `langgraph.json` once, before anyone builds the fixture. If the control plane answers with 409, the developer records the answer in this design. ## Resolved Questions @@ -170,6 +170,4 @@ None. ## Open Questions -### q01 - A second workspace, a second test deployment and a paid OpenAI key - -Without them, the tests import into the existing test workspace and deployment, never cover a move between environments, and skip the structured output check. With them, the tests copy across environments and CI runs each imported structured prompt once, billed per run. May I create a second LangSmith workspace and a second test deployment, and use a paid OpenAI key in CI? +None. diff --git a/openspec/changes/export-import-prompts-and-assistants/proposal.md b/openspec/changes/export-import-prompts-and-assistants/proposal.md index 5c7ec6f60..d22bcb43a 100644 --- a/openspec/changes/export-import-prompts-and-assistants/proposal.md +++ b/openspec/changes/export-import-prompts-and-assistants/proposal.md @@ -33,7 +33,7 @@ A prompt belongs to a workspace and an assistant belongs to a deployment, but bo - SDK: `sdk/src/prompts.rs` gains the prompt calls export and import need: a commit read that asks for the model and returns it on `CommitManifestResponse`, a commit create that sends a description and returns the new commit's `id`, a commit list, calls that list, create, move and delete commit tags, and a repo update. `Prompt` gains the repo's readme, tags and `tenant_id`, and export fills `source.workspace_id` from `tenant_id`. `sdk/src/assistants.rs` gains the `context` and `version` fields, `assistant_id` and `if_exists` on create, `graph_id` on update, and a search by graph ID. Developers commit these fields as `✨ feat(sdk)` with no `BREAKING CHANGE:` footer, so the Prepare Release workflow makes the next release a minor version. [design.md's decision on the SDK release](design.md#developers-commit-the-new-sdk-fields-as--featsdk-so-the-next-release-is-a-minor-version) links the release docs that ask for this. - CLI: new subcommands in `cli/src/commands/prompt.rs` and `cli/src/commands/assistant.rs`, and a shared module for the export file, the destination record, comparison and the exit status. -- Tests that copy between environments need a second workspace, a second deployment and one model provider key for the structured output check, which the question under Open Questions in `design.md` asks codekiln for. Until then, the tests import into the existing test workspace and deployment and the structured output check skips. Once they exist, langstar's tests read them through `fnox.toml` and the CI workflow; the destination deployment fixture from [export-import-prompts-and-assistants Task 2.4 Add a destination deployment fixture](tasks.md#L19) creates the deployment whenever the cleanup workflow has deleted it, as the source fixture does today. Nobody knows yet whether the control plane accepts a second deployment from this repository. In [export-import-prompts-and-assistants Task 2.3 Register the test graph under a second graph ID](tasks.md#L18), a developer deploys the new `langgraph.json` once to find out, before anyone builds the fixture. +- Tests that copy between environments need a second workspace, a second deployment and one model provider key for the structured output check, and the structured output check skips without the key. The tests run in the single existing workspace and test deployment until trying export and import on real assistants and prompts shows whether they need a second workspace and deployment. Once they exist, langstar's tests read them through `fnox.toml` and the CI workflow; the destination deployment fixture from [export-import-prompts-and-assistants Task 2.4 Add a destination deployment fixture](tasks.md#L19) creates the deployment whenever the cleanup workflow has deleted it, as the source fixture does today. Nobody knows yet whether the control plane accepts a second deployment from this repository. In [export-import-prompts-and-assistants Task 2.3 Register the test graph under a second graph ID](tasks.md#L18), a developer deploys the new `langgraph.json` once to find out, before anyone builds the fixture. - Docs: a new usage page on moving prompts and assistants between environments. ## Citations diff --git a/openspec/changes/export-import-prompts-and-assistants/tasks.md b/openspec/changes/export-import-prompts-and-assistants/tasks.md index 522507405..5c6a9121f 100644 --- a/openspec/changes/export-import-prompts-and-assistants/tasks.md +++ b/openspec/changes/export-import-prompts-and-assistants/tasks.md @@ -1,11 +1,11 @@ # Tasks -Each numbered group below is one pull request, merged in the order listed, except group 2, whose tasks each get a pull request of their own. Tasks 2.3 and 2.4 merge in that order: the fixture in 2.4 deploys from `main`, so the `langgraph.json` from 2.3 has to be on `main` first. Task 2.2 can merge at any time. Tasks 2.1, 2.3 and 2.4 wait on codekiln's answer to the question under Open Questions in `design.md`. Until then, the tests import into the existing test workspace and deployment, as `design.md`'s decision on test environments says, and the structured output check in 7.3 skips. +Each numbered group below is one pull request, merged in the order listed, except group 2, whose tasks each get a pull request of their own. Tasks 2.3 and 2.4 merge in that order: the fixture in 2.4 deploys from `main`, so the `langgraph.json` from 2.3 has to be on `main` first. Task 2.2 can merge at any time. Tasks 2.1, 2.3 and 2.4 apply only if the tests need a second workspace and deployment, as `design.md`'s decision on test environments says. Until then, the tests import into the existing test workspace and deployment, and the structured output check in 7.3 skips without an OpenAI key. ## 1. Check the API round trip before building on it - [ ] 1.1 Rerun the jq query for `prompt-endpoints.json` in `reference/api-specs/langsmith/FRAGMENTS.md`, so the extract picks up the `include` parameter of the commit read that its 2025-11-29 copy lacks, and extract the workspace secrets endpoints (`/api/v1/workspaces/current/secrets`) from `reference/openapi/langchain/langsmith/openapi.json` into a new file in `reference/api-specs/langsmith/` with its jq query in `FRAGMENTS.md`; verify by rerunning each query and getting the same file -- [ ] 1.2 Write `reference/research/807-openapi-validation.md`, recording what the live API returns. Verify that every claim in the report names the request that produced it. This task pushes to the second workspace and runs a prompt with a model, so it waits on the destination workspace and the OpenAI key that codekiln creates for task 2.1. +- [ ] 1.2 Write `reference/research/807-openapi-validation.md`, recording what the live API returns. Verify that every claim in the report names the request that produced it. Without a second workspace, this task pushes to the existing test workspace under a fresh handle, and it runs a prompt with a model only when an OpenAI key is set. - For a structured prompt saved with a model, read with `include=model`: the manifest, its outer `id`, the `model_config` and `model_provider` beside it, and whether `model_provider` matches the class name export writes to `requires.model_class`. - Push that manifest to a second workspace: whether the read-back returns the same manifest, `model_config` and `model_provider`, and what pushing an identical manifest again returns. - On the Agent Server: what `POST /assistants/search` returns for a graph ID the deployment does not serve, what it returns for `context`, whether an assistant update that sends a metadata key with a null value removes that key, whether an update that sends `config` or `context` replaces the assistant's object or merges into it, and whether an update can clear the description. @@ -13,7 +13,7 @@ Each numbered group below is one pull request, merged in the order listed, excep ## 2. Test environments -- [ ] 2.1 Add the destination workspace and the model provider key to `fnox.toml`, the CI workflow and `docs/dev/environment-variables.md`, with placeholders only; verify with `mise run check` and a CI run that reads both. This task waits on codekiln to create a second workspace in the test organization and an OpenAI API key, and to store both in 1Password and the repository's CI secrets and the key as the destination workspace's `OPENAI_API_KEY` secret. The import tests need the workspace as a place to import into, and its secret lets prompt import's secret check pass for a prompt with an OpenAI model. The key lets the check in 7.3 run each imported structured prompt once against a small OpenAI model, which bills that account on every CI run; it is the one test that shows a copy still returns structured data, where every other test compares text. +- [ ] 2.1 Add the destination workspace and the model provider key to `fnox.toml`, the CI workflow and `docs/dev/environment-variables.md`, with placeholders only; verify with `mise run check` and a CI run that reads both. This task applies only if the tests need a second workspace, and adds the key only once CI has one. The import tests need the workspace as a place to import into, and its secret lets prompt import's secret check pass for a prompt with an OpenAI model. The key lets the check in 7.3 run each imported structured prompt once against a small OpenAI model, which bills that account on every CI run; it is the one test that shows a copy still returns structured data, where every other test compares text. - [ ] 2.2 Replace the bare `#[ignore]` on `test_assistant_create_basic` in `cli/tests/assistant_command_test.rs`, whose reason says a deployment built from GitHub has no `custom_url`, with `#[cfg_attr(not(feature = "integration-tests"), ignore)]`, the guard `test_assistant_description_field` beside it carries. The control plane sets `source_config.custom_url` on a deployment built from GitHub, as `SourceConfig.custom_url` in `reference/openapi/langchain/control-plane/openapi.json` says. Verify by running the test against the test deployment. - [ ] 2.3 Add a second `langgraph.json` under `tests/fixtures/` that registers the test graph from `tests/fixtures/test-graph-deployment/langgraph.json` under two graph IDs, `test_graph` and `test_graph_alt`, so the update tests in 3.3 and 10.2 can move an assistant from one graph to the other. To verify, deploy the new file while the source test deployment is live, which shows whether the control plane accepts a second deployment from this repository: run `langstar deployment create` with `--branch` set to this pull request's branch and `--config-path` set to the new file. A 409 "A deployment already exists for this agent environment" means it does not, and the developer reports that to codekiln before anyone starts task 2.4. Otherwise run `langstar assistant list --deployment `, find one assistant with graph ID `test_graph` and one with `test_graph_alt`, then delete the deployment. - [ ] 2.4 Add a destination deployment fixture to `sdk/src/test_utils.rs` and `cli/tests/common/fixtures.rs` that deploys the `langgraph.json` from 2.3 under the name prefix `dest-integration-test`. `find_reusable_deployment` returns any deployment whose name starts with the fixture's prefix before it checks the graph config, so this prefix keeps the source fixture, whose prefix is `pr-integration-test`, from picking up the destination deployment. `find_reusable_deployment` then falls back to any live deployment built from the same repository and graph config file, so the separate file from 2.3 keeps the destination fixture from picking up the source deployment. The control plane answers a second deployment of one agent environment with 409 "A deployment already exists for this agent environment". No document says which fields make an agent environment, and the verification in 2.3 shows whether a second graph config file in the same repository makes a new one. `.github/workflows/cleanup-test-deployments.yml` runs every four hours and deletes each deployment whose name contains `integration-test` once it is four hours old or older, so the destination deployment is deleted and rebuilt on the same schedule as the source deployment. Verify with an integration test that asserts the destination and source deployment IDs differ and lists assistants on the destination deployment with `langstar assistant list` From f037b0e270771af43d3d805d59f68577f5ac708c Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Tue, 27 Oct 2026 17:46:12 -0400 Subject: [PATCH 78/90] =?UTF-8?q?=F0=9F=93=9A=20docs(openspec):=20run=20th?= =?UTF-8?q?e=20structured=20output=20check=20from=20a=20committed=20uv=20s?= =?UTF-8?q?cript=20lockfile?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 5.5 --- openspec/changes/export-import-prompts-and-assistants/design.md | 2 +- openspec/changes/export-import-prompts-and-assistants/tasks.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/openspec/changes/export-import-prompts-and-assistants/design.md b/openspec/changes/export-import-prompts-and-assistants/design.md index c0e55115a..4bbfc0ba3 100644 --- a/openspec/changes/export-import-prompts-and-assistants/design.md +++ b/openspec/changes/export-import-prompts-and-assistants/design.md @@ -122,7 +122,7 @@ Langstar exits today only with 0, with 1 from `main` in `cli/src/main.rs` when a ### An optional Python script checks that an imported structured prompt still returns structured data -The LangSmith API has no documented way to run a prompt with its model, so a short Python script runs each imported structured prompt with LangChain and checks that the result matches the prompt's schema. The one prompt endpoint in the OpenAPI spec, `POST /api/v1/prompts/invoke_prompt`, takes message templates and inputs with no model or schema, and no LangSmith SDK calls it; in [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8), a developer calls it on the live API in case it does more than the spec says. A mise task runs the script with `uv run --script`, which installs the `langsmith` and `langchain-openai` packages the script lists, on a developer's machine and in the Integration Tests CI job. The script reads the destination workspace: it builds its LangSmith `Client` with `workspace_id` set to `LANGSTAR_TEST_DEST_WORKSPACE_ID`, because both `fnox.toml` and CI set `LANGSMITH_WORKSPACE_ID` to the source workspace, which holds a prompt with the same handle. The script pulls the imported prompt with `include_model=True` and passes `secrets={"OPENAI_API_KEY": }` to [`pull_prompt`](https://github.com/langchain-ai/langsmith-sdk/blob/47c553caf08c1989e2be3ff1886da3eb801f709f/python/langsmith/client.py#L10228), because the LangSmith Python SDK reads no secret from the environment unless asked, and raises when a manifest holds a secret reference and no `secrets` map arrives. The script needs an OpenAI API key, which bills on every run. The check is optional: the mise task skips it and says so when `OPENAI_API_KEY` is unset, so every other test runs without a paid key, and a developer with a key of their own can run it by hand. +The LangSmith API has no documented way to run a prompt with its model, so a short Python script runs each imported structured prompt with LangChain and checks that the result matches the prompt's schema. The one prompt endpoint in the OpenAPI spec, `POST /api/v1/prompts/invoke_prompt`, takes message templates and inputs with no model or schema, and no LangSmith SDK calls it; in [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8), a developer calls it on the live API in case it does more than the spec says. A mise task runs the script with `uv run --locked --script` against a committed script lockfile, which installs the `langsmith` and `langchain-openai` packages the script lists, on a developer's machine and in the Integration Tests CI job. The script reads the destination workspace: it builds its LangSmith `Client` with `workspace_id` set to `LANGSTAR_TEST_DEST_WORKSPACE_ID`, because both `fnox.toml` and CI set `LANGSMITH_WORKSPACE_ID` to the source workspace, which holds a prompt with the same handle. The script pulls the imported prompt with `include_model=True` and passes `secrets={"OPENAI_API_KEY": }` to [`pull_prompt`](https://github.com/langchain-ai/langsmith-sdk/blob/47c553caf08c1989e2be3ff1886da3eb801f709f/python/langsmith/client.py#L10228), because the LangSmith Python SDK reads no secret from the environment unless asked, and raises when a manifest holds a secret reference and no `secrets` map arrives. The script needs an OpenAI API key, which bills on every run. The check is optional: the mise task skips it and says so when `OPENAI_API_KEY` is unset, so every other test runs without a paid key, and a developer with a key of their own can run it by hand. ### Export and import use the same SDK calls as the planned `prompt create`, `get` and `update` diff --git a/openspec/changes/export-import-prompts-and-assistants/tasks.md b/openspec/changes/export-import-prompts-and-assistants/tasks.md index 5c6a9121f..6283aa7dd 100644 --- a/openspec/changes/export-import-prompts-and-assistants/tasks.md +++ b/openspec/changes/export-import-prompts-and-assistants/tasks.md @@ -109,7 +109,7 @@ Each integration test in this group carries `#[cfg_attr(not(feature = "integrati - An integration test that `--update` against a destination whose latest commit matches the file and carries a `staging` tag the file lacks deletes `staging`, adds no commit and exits with status 0, and that `--dry-run` lists the delete. - An integration test that `--dry-run` writes nothing. - A unit test in `cli/src/commands/prompt.rs` that serves, through a `mockito` server and `LangchainClient::with_base_urls`, a commit read back that lacks a field the file holds, calls the import code, and asserts that the result names that field's JSON Pointer path and gives exit status 3, and that the destination record holds the new commit hash. -- [ ] 7.3 Add the structured output check as a mise task that runs a Python script with `uv run --script`. The script names `langsmith`, `langchain-openai` and `jsonschema` as inline script dependencies, each pinned to an exact version, so a new upstream release cannot change the CI result, and validates each result with `jsonschema` against the prompt's schema. It builds its `Client` with `workspace_id` set to `LANGSTAR_TEST_DEST_WORKSPACE_ID`, because both `fnox.toml` and CI set `LANGSMITH_WORKSPACE_ID` to the source workspace, which holds a prompt with the same handle. It then pulls the imported prompt with `include_model=True` and a `secrets` map holding the model provider key read from the environment, invokes it and validates the result against the schema. Add `uv` to `mise.toml`, and remove `install: false` from the Setup mise step of the Integration Tests job in `.github/workflows/ci.yml` so that job installs `uv` and Python 3.11. Verify by having the import integration test create a `json_schema` prompt and a `function_calling` prompt, each saved with a small OpenAI model, import both into the destination workspace and run the check on each, under `mise run check` and in that CI job. When `OPENAI_API_KEY` is unset, the task prints that it skipped the check and exits with status 0; verify that too, by running it without the key. +- [ ] 7.3 Add the structured output check as a mise task that runs a Python script with `uv run --locked --script`. The script names `langsmith`, `langchain-openai` and `jsonschema` as inline script dependencies, each pinned to an exact version. Commit the lockfile `uv lock --script` writes next to the script, so `--locked` fixes every transitive dependency too and a new upstream release cannot change the CI result. The script validates each result with `jsonschema` against the prompt's schema. It builds its `Client` with `workspace_id` set to `LANGSTAR_TEST_DEST_WORKSPACE_ID`, because both `fnox.toml` and CI set `LANGSMITH_WORKSPACE_ID` to the source workspace, which holds a prompt with the same handle. It then pulls the imported prompt with `include_model=True` and a `secrets` map holding the model provider key read from the environment, invokes it and validates the result against the schema. Add `uv` to `mise.toml`, and remove `install: false` from the Setup mise step of the Integration Tests job in `.github/workflows/ci.yml` so that job installs `uv` and Python 3.11. Verify by having the import integration test create a `json_schema` prompt and a `function_calling` prompt, each saved with a small OpenAI model, import both into the destination workspace and run the check on each, under `mise run check` and in that CI job. When `OPENAI_API_KEY` is unset, the task prints that it skipped the check and exits with status 0; verify that too, by running it without the key. - [ ] 7.4 Document prompt import, the workspace scope of prompts and setting up destination credentials; verify that each documented command runs as written ## 8. assistant export From 7850530475f49eb6374652855d6f3a04741899d3 Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Tue, 27 Oct 2026 17:54:25 -0400 Subject: [PATCH 79/90] =?UTF-8?q?=F0=9F=93=9A=20docs(openspec):=20record?= =?UTF-8?q?=20the=20export=20and=20compare=20trial=20in=20the=20transfer?= =?UTF-8?q?=20design?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Decision 8 now states the trial's result: one workspace and one test deployment are enough, and a local langgraph dev server can serve as an import target. Add scenarios and tasks for refusing a graph's default assistant, the -/ owner in prompt names and exporting a private prompt. Co-Authored-By: Claude Opus 5.5 --- .../design.md | 2 +- .../specs/assistant/spec.md | 20 +++++++++++++++++++ .../specs/prompt/spec.md | 6 ++++++ .../tasks.md | 4 +++- 4 files changed, 30 insertions(+), 2 deletions(-) diff --git a/openspec/changes/export-import-prompts-and-assistants/design.md b/openspec/changes/export-import-prompts-and-assistants/design.md index 4bbfc0ba3..7889135b0 100644 --- a/openspec/changes/export-import-prompts-and-assistants/design.md +++ b/openspec/changes/export-import-prompts-and-assistants/design.md @@ -150,7 +150,7 @@ The prompt's manifest carries the model and its settings, so the prompt runs wit ### The tests run in the single existing workspace and test deployment -The integration tests import each prompt into the existing test workspace under a fresh handle, and each assistant onto the existing test deployment as a new assistant. That covers import, compare, verify and the destination record, but not a move between workspaces or deployments, and the test that moves an assistant to `test_graph_alt` is skipped. Whether the tests need a second workspace and deployment will be settled by trying export and import on real assistants and prompts before implementation, and the result will be recorded here. +The integration tests import each prompt into the existing test workspace under a fresh handle, and each assistant onto the existing test deployment as a new assistant. That covers import, compare, verify and the destination record, but not a move between workspaces or deployments, and the test that moves an assistant to `test_graph_alt` is skipped. Trying export and compare on a real set of assistants and prompts, with langstar's current commands and the SDK, showed that one workspace and one test deployment are enough. Only the assistants needed to move; the prompts stayed in one shared workspace, and the assistants name them without an owner or a commit. Every failure the trial found shows up in an export, or in an import to any deployment, including a local `langgraph dev` server, so a local `langgraph dev` server can serve as an import target. The trial needed no live model call either: comparing each structured prompt's schema and model settings finds a bad copy, and a live call tests the model, not the copy. ## Risks / Trade-offs diff --git a/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md b/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md index c5733b2f7..4673f5b9a 100644 --- a/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md +++ b/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md @@ -21,6 +21,15 @@ - **THEN** stdout holds only the export file's JSON document - **AND** any progress message goes to stderr +### Requirement: Refuse to export a graph's default assistant + +`langstar assistant export` SHALL refuse a graph's default assistant, which every deployment creates for each graph with the same ID. + +#### Scenario: Assistant is a graph's default assistant + +- **WHEN** the assistant is the default assistant the deployment created for its graph, with `created_by` set to `system` in its metadata +- **THEN** langstar writes no file, says that every deployment creates that assistant, and exits with status 1 + ### Requirement: Refuse to export an assistant that holds a credential Before it writes, `langstar assistant export` SHALL run the credential check over every string at any depth of the `assistant` object, skipping secret references and empty strings. When the check flags a string, export SHALL write no file and nothing to stdout, list the JSON Pointer path and rule of each flagged string, without the string or any part of it, and exit with status 1. Export SHALL NOT write a file with `` in place of a flagged string. @@ -107,6 +116,17 @@ The credential check SHALL flag a string, under any key, that holds a bearer tok - **WHEN** the pointer resolves to no value in the assistant - **THEN** langstar exits with status 1 and an error naming the pointer +### Requirement: Read the owner `-` as the current workspace + +A prompt name under `requires.prompts` with the owner `-`, such as `-/support-system:prod`, SHALL mean the current workspace's prompt, as a name without an owner does. + +#### Scenario: Assistant names a private prompt with the owner `-` + +- **WHEN** a user runs `langstar assistant export --deployment my-agent --prompt-ref /config/configurable/system_prompt` +- **AND** that value is `-/support-system:prod` +- **THEN** `requires.prompts` holds the pointer and `-/support-system:prod` +- **AND** `langstar assistant verify` checks the `prod` tag of the workspace's `support-system` prompt + ### Requirement: Import an assistant into a deployment With no created assistant ID recorded for the deployment and no `--assistant-id`, `langstar assistant import --deployment ` SHALL choose a new assistant ID, write it as pending in the destination record, then create the assistant with that `assistant_id`, `if_exists: do_nothing` and the file's name, description, graph ID, configuration, context and metadata, mark the entry created, and print the ID. A pending entry SHALL make the next import repeat the create. diff --git a/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md index b25fd1e3d..0bd7ab355 100644 --- a/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md +++ b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md @@ -25,6 +25,12 @@ - **WHEN** no prompt in the workspace has the handle - **THEN** langstar exits with status 1 and an error naming the handle +#### Scenario: Export a private prompt + +- **WHEN** the workspace's `my-prompt` is private +- **AND** a user runs `langstar prompt export my-prompt --output my-prompt.json` or `langstar prompt export -/my-prompt --output my-prompt.json` +- **THEN** langstar reads the commit as `-/my-prompt` and writes `my-prompt.json` + #### Scenario: Export to stdout - **WHEN** a user runs `langstar prompt export my-prompt` without `--output` diff --git a/openspec/changes/export-import-prompts-and-assistants/tasks.md b/openspec/changes/export-import-prompts-and-assistants/tasks.md index 6283aa7dd..7f664e530 100644 --- a/openspec/changes/export-import-prompts-and-assistants/tasks.md +++ b/openspec/changes/export-import-prompts-and-assistants/tasks.md @@ -58,6 +58,7 @@ Each integration test in this group carries `#[cfg_attr(not(feature = "integrati - Integration tests that create four prompts through the SDK (a `json_schema` prompt with no model, a `function_calling` prompt with a nested schema, a prompt with several messages and a placeholder, and a prompt with a model and metadata), export each through the CLI, and assert every field of the file against the SDK's read of the commit. The file for the prompt with a model lists `OPENAI_API_KEY` under `requires.secrets` and `ChatOpenAI` under `requires.model_class`, and holds no secret value. The file for the `json_schema` prompt holds an empty `requires.secrets` and null under `requires.model_class`. - An integration test that exports, with `--commit prod`, a prompt whose `prod` tag points at an earlier commit, and asserts that `source.commit_hash` and the manifest are that earlier commit's. - An error test that exports a handle the workspace has no prompt for, and asserts that it exits with status 1, names the handle and writes no file. + - An integration test that exports a private prompt once as `my-prompt` and once as `-/my-prompt`, without `--`, and asserts that both files are identical. - A CLI test that runs `prompt export` without `--output`, parses all of stdout as one JSON document, and asserts that its `source.commit_hash` matches the SDK's read and that any progress message is on stderr. - An integration test that gives one commit two tags, exports it twice, and asserts that the two files are identical. - Unit tests in `cli/src/commands/prompt.rs` that serve, through a `mockito` server and `LangchainClient::with_base_urls`, one commit for each export refusal scenario in `specs/prompt/spec.md`: a `ChatOpenAI` constructor with a string under `openai_api_key`, and another under `OPENAI_API_KEY`; `default_headers.Authorization` holding `Bearer ` and a token; `model_kwargs.extra` holding a string that starts with `sk-`; manifest `metadata.source_db` holding a `postgresql://` URL with a password; and a repo readme holding a PEM private key. Each calls the export code with `--output my-prompt.json`, and once without `--output`, and asserts that it exits with status 1, names the path and the rule without the string, writes no file and prints nothing to stdout. @@ -120,6 +121,7 @@ Each integration test in this group carries `#[cfg_attr(not(feature = "integrati - An integration test that exports the same assistant version twice and asserts that the two files are identical. - An error test for a pointer that names nothing. - An error test that exporting an assistant ID the deployment lacks exits with status 1, names the ID and writes no file. + - An integration test that exporting a graph's default assistant on the test deployment exits with status 1 and writes no file. - Unit tests in `cli/src/commands/assistant.rs` that serve, through a `mockito` server and `LangchainClient::with_base_urls`, one assistant for each export refusal scenario in `specs/assistant/spec.md`: `config.configurable.OPENAI_API_KEY`, `context.anthropic_api_key` and `metadata.openai_api_key` holding a string; `context.github_access_token` holding a string; `config.configurable.db.password` holding a string; `config.configurable.checkpoint_uri` holding a `postgresql://` URL with a password; `config.configurable.tool_headers.Authorization` holding `Bearer ` and a token; and `metadata.deploy_note` holding an OpenSSH private key. Each calls the export code with `--output support-bot.json`, and once without `--output`, and asserts that it exits with status 1, names the JSON Pointer path and the rule without the string, writes no file and prints nothing to stdout. - A unit test that serves an assistant whose `config.configurable.openai_api_key` holds a secret reference, and asserts that export writes the file with the reference. - A unit test that exports an assistant whose `config.configurable.token_budget_policy` holds `strict` with `--not-a-secret /config/configurable/token_budget_policy`, and asserts that the file holds `strict` there and stderr names the path, and one without the flag that asserts status 1. @@ -139,7 +141,7 @@ Each integration test in this group carries `#[cfg_attr(not(feature = "integrati - An integration test of a file whose graph ID the deployment lacks, which reports the graph ID and exits with status 3. - Unit tests in `cli/src/commands/assistant.rs` that serve, through a `mockito` server, a search that answers 422 with an error naming `graph_id`, and assert that verify reports the graph ID missing and exits with status 3, and a 422 naming another field, and assert that verify exits with status 1. - An integration test of a file whose prompt tag the workspace lacks, which reports the prompt and exits with status 3. - - A unit test that a prompt reference with no owner and no tag, such as `support-system`, makes the prompt check request `GET /api/v1/commits/-/support-system/latest`, and one that `my-org/support-system:prod` requests `GET /api/v1/commits/my-org/support-system/prod`. + - A unit test that a prompt reference with no owner and no tag, such as `support-system`, makes the prompt check request `GET /api/v1/commits/-/support-system/latest`, one that `my-org/support-system:prod` requests `GET /api/v1/commits/my-org/support-system/prod`, and one that `-/support-system:prod` requests `GET /api/v1/commits/-/support-system/prod`. - An integration test that runs verify with `--workspace-id` set to the destination workspace, for an assistant whose referenced prompt tag exists only in that workspace, and asserts that verify exits with status 0 and that `checks` reports the prompt as found. - A CLI test that `langstar -f json assistant verify` prints one JSON object with `status`, `differences` and `checks` to stdout. - [ ] 9.3 Document assistant compare and verify; verify that each documented command runs as written From f45ecc893417eba2d46d18128c1a4e33a0263e32 Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Wed, 28 Oct 2026 12:33:00 -0400 Subject: [PATCH 80/90] =?UTF-8?q?=F0=9F=93=9A=20docs(openspec):=20run=20th?= =?UTF-8?q?e=20transfer=20tests=20in=20one=20workspace=20until=20a=20secon?= =?UTF-8?q?d=20one=20exists?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Copilot pointed out that the tasks called the second workspace and deployment optional while task 3.3 needed the destination fixture from 2.4, and task 7.3 read LANGSTAR_TEST_DEST_WORKSPACE_ID, which only 2.1 sets. Until those tasks merge, the destination workspace and deployment are the existing test ones. The graph moves in 3.3 and 10.2 and the destination-only prompt in 9.2 carry an ignore naming the task that adds their environment, and the structured output script takes the workspace ID the import test used. Co-Authored-By: Claude Opus 5.5 --- .../export-import-prompts-and-assistants/design.md | 4 ++-- .../export-import-prompts-and-assistants/tasks.md | 10 +++++----- 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/openspec/changes/export-import-prompts-and-assistants/design.md b/openspec/changes/export-import-prompts-and-assistants/design.md index 7889135b0..2e485e54a 100644 --- a/openspec/changes/export-import-prompts-and-assistants/design.md +++ b/openspec/changes/export-import-prompts-and-assistants/design.md @@ -122,7 +122,7 @@ Langstar exits today only with 0, with 1 from `main` in `cli/src/main.rs` when a ### An optional Python script checks that an imported structured prompt still returns structured data -The LangSmith API has no documented way to run a prompt with its model, so a short Python script runs each imported structured prompt with LangChain and checks that the result matches the prompt's schema. The one prompt endpoint in the OpenAPI spec, `POST /api/v1/prompts/invoke_prompt`, takes message templates and inputs with no model or schema, and no LangSmith SDK calls it; in [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8), a developer calls it on the live API in case it does more than the spec says. A mise task runs the script with `uv run --locked --script` against a committed script lockfile, which installs the `langsmith` and `langchain-openai` packages the script lists, on a developer's machine and in the Integration Tests CI job. The script reads the destination workspace: it builds its LangSmith `Client` with `workspace_id` set to `LANGSTAR_TEST_DEST_WORKSPACE_ID`, because both `fnox.toml` and CI set `LANGSMITH_WORKSPACE_ID` to the source workspace, which holds a prompt with the same handle. The script pulls the imported prompt with `include_model=True` and passes `secrets={"OPENAI_API_KEY": }` to [`pull_prompt`](https://github.com/langchain-ai/langsmith-sdk/blob/47c553caf08c1989e2be3ff1886da3eb801f709f/python/langsmith/client.py#L10228), because the LangSmith Python SDK reads no secret from the environment unless asked, and raises when a manifest holds a secret reference and no `secrets` map arrives. The script needs an OpenAI API key, which bills on every run. The check is optional: the mise task skips it and says so when `OPENAI_API_KEY` is unset, so every other test runs without a paid key, and a developer with a key of their own can run it by hand. +The LangSmith API has no documented way to run a prompt with its model, so a short Python script runs each imported structured prompt with LangChain and checks that the result matches the prompt's schema. The one prompt endpoint in the OpenAPI spec, `POST /api/v1/prompts/invoke_prompt`, takes message templates and inputs with no model or schema, and no LangSmith SDK calls it; in [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8), a developer calls it on the live API in case it does more than the spec says. A mise task runs the script with `uv run --locked --script` against a committed script lockfile, which installs the `langsmith` and `langchain-openai` packages the script lists, on a developer's machine and in the Integration Tests CI job. The script reads the workspace import wrote to: the import test passes it the workspace ID it gave `prompt import --workspace-id` and the handle it imported under, and the script builds its LangSmith `Client` with `workspace_id` set to that ID rather than reading `LANGSMITH_WORKSPACE_ID`. That is the existing test workspace today, and the destination workspace once [export-import-prompts-and-assistants Task 2.1 Add the destination workspace and the model provider key](tasks.md#L16) adds one. The script pulls the imported prompt with `include_model=True` and passes `secrets={"OPENAI_API_KEY": }` to [`pull_prompt`](https://github.com/langchain-ai/langsmith-sdk/blob/47c553caf08c1989e2be3ff1886da3eb801f709f/python/langsmith/client.py#L10228), because the LangSmith Python SDK reads no secret from the environment unless asked, and raises when a manifest holds a secret reference and no `secrets` map arrives. The script needs an OpenAI API key, which bills on every run. The check is optional: the mise task skips it and says so when `OPENAI_API_KEY` is unset, so every other test runs without a paid key, and a developer with a key of their own can run it by hand. ### Export and import use the same SDK calls as the planned `prompt create`, `get` and `update` @@ -150,7 +150,7 @@ The prompt's manifest carries the model and its settings, so the prompt runs wit ### The tests run in the single existing workspace and test deployment -The integration tests import each prompt into the existing test workspace under a fresh handle, and each assistant onto the existing test deployment as a new assistant. That covers import, compare, verify and the destination record, but not a move between workspaces or deployments, and the test that moves an assistant to `test_graph_alt` is skipped. Trying export and compare on a real set of assistants and prompts, with langstar's current commands and the SDK, showed that one workspace and one test deployment are enough. Only the assistants needed to move; the prompts stayed in one shared workspace, and the assistants name them without an owner or a commit. Every failure the trial found shows up in an export, or in an import to any deployment, including a local `langgraph dev` server, so a local `langgraph dev` server can serve as an import target. The trial needed no live model call either: comparing each structured prompt's schema and model settings finds a bad copy, and a live call tests the model, not the copy. +The integration tests import each prompt into the existing test workspace under a fresh handle, and each assistant onto the existing test deployment as a new assistant. That covers import, compare, verify and the destination record, but not a move between workspaces or deployments. The tests that need a second environment carry an `#[ignore]` naming the task that adds it: the assistant graph moves to `test_graph_alt` in [export-import-prompts-and-assistants Task 3.3 Add the assistant fields to the SDK](tasks.md#L29) and [export-import-prompts-and-assistants Task 10.2 Add assistant import --update](tasks.md#L158) need the destination deployment from [export-import-prompts-and-assistants Task 2.4 Add a destination deployment fixture](tasks.md#L19), and the test in [export-import-prompts-and-assistants Task 9.2 Add assistant verify](tasks.md#L138) of a prompt that only the destination workspace holds needs the workspace from [export-import-prompts-and-assistants Task 2.1 Add the destination workspace and the model provider key](tasks.md#L16). Trying export and compare on a real set of assistants and prompts, with langstar's current commands and the SDK, showed that one workspace and one test deployment are enough. Only the assistants needed to move; the prompts stayed in one shared workspace, and the assistants name them without an owner or a commit. Every failure the trial found shows up in an export, or in an import to any deployment, including a local `langgraph dev` server, so a local `langgraph dev` server can serve as an import target. The trial needed no live model call either: comparing each structured prompt's schema and model settings finds a bad copy, and a live call tests the model, not the copy. ## Risks / Trade-offs diff --git a/openspec/changes/export-import-prompts-and-assistants/tasks.md b/openspec/changes/export-import-prompts-and-assistants/tasks.md index 7f664e530..75c57ef6b 100644 --- a/openspec/changes/export-import-prompts-and-assistants/tasks.md +++ b/openspec/changes/export-import-prompts-and-assistants/tasks.md @@ -1,6 +1,6 @@ # Tasks -Each numbered group below is one pull request, merged in the order listed, except group 2, whose tasks each get a pull request of their own. Tasks 2.3 and 2.4 merge in that order: the fixture in 2.4 deploys from `main`, so the `langgraph.json` from 2.3 has to be on `main` first. Task 2.2 can merge at any time. Tasks 2.1, 2.3 and 2.4 apply only if the tests need a second workspace and deployment, as `design.md`'s decision on test environments says. Until then, the tests import into the existing test workspace and deployment, and the structured output check in 7.3 skips without an OpenAI key. +Each numbered group below is one pull request, merged in the order listed, except group 2, whose tasks each get a pull request of their own. Tasks 2.3 and 2.4 merge in that order: the fixture in 2.4 deploys from `main`, so the `langgraph.json` from 2.3 has to be on `main` first. Task 2.2 can merge at any time. Tasks 2.1, 2.3 and 2.4 apply only if the tests need a second workspace and deployment, as `design.md`'s decision on test environments says. Until they merge, a task below that names the destination workspace means the existing test workspace from `LANGSMITH_WORKSPACE_ID`, into which each test imports a prompt under a fresh handle, and a task that names the destination deployment means the existing test deployment. The tests that need a second environment carry `#[ignore = "needs task 2.1"]` or `#[ignore = "needs task 2.4"]`, and the pull request for that task removes the attribute: the graph moves in 3.3 and 10.2 need `test_graph_alt` on the destination deployment from 2.4, and the prompt in 9.2 that exists only in the destination workspace needs the workspace from 2.1. The structured output check in 7.3 skips without an OpenAI key. ## 1. Check the API round trip before building on it @@ -26,7 +26,7 @@ Each integration test in this group carries `#[cfg_attr(not(feature = "integrati - [ ] 3.1 Add `description`, `model_config` and `model_provider` to `CommitManifestResponse`, and a commit read that sends `include=model` and returns that struct, to `sdk/src/prompts.rs`; verify with a `mockito` test that asserts the query string and the parsed fields, and an integration test that reads a prompt with a model - [ ] 3.2 Add the commit's `id` to `CommitData` in `sdk/src/prompts.rs`, a call that lists a prompt's commits with their `id` and hash (`GET /api/v1/commits/{owner}/{repo}`), calls to list, create and move commit tags, which name their commit by that `id` in `POST /api/v1/repos/{owner}/{repo}/tags` and `PATCH /api/v1/repos/{owner}/{repo}/tags/{tag_name}`, and a call that deletes a tag with `DELETE /api/v1/repos/{owner}/{repo}/tags/{tag_name}`, which `prompt import --update` from 7.2 and the end-to-end test in 10.3 use; verify with `mockito` tests that assert `commit_id` in each create and move request body and the path of the delete request, and an integration test that pushes two commits, tags the first, finds the second commit's `id` through the commit list, moves the tag to it, reads the tag back through the SDK, deletes it, and asserts that the tag list no longer holds it -- [ ] 3.3 Add `context` and `version` to `Assistant`, `context` and `graph_id` to `UpdateAssistantRequest`, `context`, `assistant_id` and `if_exists` to `CreateAssistantRequest`, and a `graph_id` filter to `AssistantSearchRequest`, in `sdk/src/assistants.rs`; verify with `mockito` tests of the request bodies and an integration test that creates an assistant with context on the destination deployment from 2.4, reads it back with its version, finds it by graph ID, moves it from `test_graph` to `test_graph_alt`, and reads `test_graph_alt` back. The test runs on the destination deployment because the source deployment serves only `test_graph`, and task 2.4, which adds the destination fixture, merges before this group. +- [ ] 3.3 Add `context` and `version` to `Assistant`, `context` and `graph_id` to `UpdateAssistantRequest`, `context`, `assistant_id` and `if_exists` to `CreateAssistantRequest`, and a `graph_id` filter to `AssistantSearchRequest`, in `sdk/src/assistants.rs`; verify with `mockito` tests of the request bodies, an integration test that creates an assistant with context on the existing test deployment, reads it back with its version and finds it by graph ID, and a second integration test that creates an assistant on the destination deployment from 2.4, moves it from `test_graph` to `test_graph_alt`, and reads `test_graph_alt` back. The second test carries `#[ignore = "needs task 2.4"]`, because the existing test deployment serves only `test_graph`. - [ ] 3.4 Add `description` to `CommitRequest`, and `readme`, the repo's `tags` and `tenant_id` to `Prompt`, which `PromptClient::get` returns and export reads the repo details and `source.workspace_id` from, and a repo update that sends `PATCH /api/v1/repos/{owner}/{repo}` with `description`, `readme`, `tags` and `is_public`, to `sdk/src/prompts.rs`; verify with `mockito` tests of the request bodies and of a `PromptClient::get` response that holds `readme`, `tags` and `tenant_id`, and an integration test that pushes a commit with a description and reads it back through the commit read from 3.1, then updates the repo's description, readme, tags and visibility and reads each back through `PromptClient::get` ## 4. Shared code for the prompt and assistant commands: the export file, the comparison and the report @@ -110,7 +110,7 @@ Each integration test in this group carries `#[cfg_attr(not(feature = "integrati - An integration test that `--update` against a destination whose latest commit matches the file and carries a `staging` tag the file lacks deletes `staging`, adds no commit and exits with status 0, and that `--dry-run` lists the delete. - An integration test that `--dry-run` writes nothing. - A unit test in `cli/src/commands/prompt.rs` that serves, through a `mockito` server and `LangchainClient::with_base_urls`, a commit read back that lacks a field the file holds, calls the import code, and asserts that the result names that field's JSON Pointer path and gives exit status 3, and that the destination record holds the new commit hash. -- [ ] 7.3 Add the structured output check as a mise task that runs a Python script with `uv run --locked --script`. The script names `langsmith`, `langchain-openai` and `jsonschema` as inline script dependencies, each pinned to an exact version. Commit the lockfile `uv lock --script` writes next to the script, so `--locked` fixes every transitive dependency too and a new upstream release cannot change the CI result. The script validates each result with `jsonschema` against the prompt's schema. It builds its `Client` with `workspace_id` set to `LANGSTAR_TEST_DEST_WORKSPACE_ID`, because both `fnox.toml` and CI set `LANGSMITH_WORKSPACE_ID` to the source workspace, which holds a prompt with the same handle. It then pulls the imported prompt with `include_model=True` and a `secrets` map holding the model provider key read from the environment, invokes it and validates the result against the schema. Add `uv` to `mise.toml`, and remove `install: false` from the Setup mise step of the Integration Tests job in `.github/workflows/ci.yml` so that job installs `uv` and Python 3.11. Verify by having the import integration test create a `json_schema` prompt and a `function_calling` prompt, each saved with a small OpenAI model, import both into the destination workspace and run the check on each, under `mise run check` and in that CI job. When `OPENAI_API_KEY` is unset, the task prints that it skipped the check and exits with status 0; verify that too, by running it without the key. +- [ ] 7.3 Add the structured output check as a mise task that runs a Python script with `uv run --locked --script`. The script names `langsmith`, `langchain-openai` and `jsonschema` as inline script dependencies, each pinned to an exact version. Commit the lockfile `uv lock --script` writes next to the script, so `--locked` fixes every transitive dependency too and a new upstream release cannot change the CI result. The script validates each result with `jsonschema` against the prompt's schema. The script takes the workspace ID and the handle as arguments, and the import integration test passes the same workspace ID it passed to `prompt import --workspace-id` and the handle it imported under. The script builds its `Client` with `workspace_id` set to that argument rather than reading `LANGSMITH_WORKSPACE_ID`, so it reads the workspace import wrote to, before and after task 2.1 adds a second one. It then pulls the imported prompt with `include_model=True` and a `secrets` map holding the model provider key read from the environment, invokes it and validates the result against the schema. Add `uv` to `mise.toml`, and remove `install: false` from the Setup mise step of the Integration Tests job in `.github/workflows/ci.yml` so that job installs `uv` and Python 3.11. Verify by having the import integration test create a `json_schema` prompt and a `function_calling` prompt, each saved with a small OpenAI model, import both into the destination workspace and run the check on each with that workspace's ID, under `mise run check` and in that CI job. When `OPENAI_API_KEY` is unset, the task prints that it skipped the check and exits with status 0; verify that too, by running it without the key. - [ ] 7.4 Document prompt import, the workspace scope of prompts and setting up destination credentials; verify that each documented command runs as written ## 8. assistant export @@ -142,7 +142,7 @@ Each integration test in this group carries `#[cfg_attr(not(feature = "integrati - Unit tests in `cli/src/commands/assistant.rs` that serve, through a `mockito` server, a search that answers 422 with an error naming `graph_id`, and assert that verify reports the graph ID missing and exits with status 3, and a 422 naming another field, and assert that verify exits with status 1. - An integration test of a file whose prompt tag the workspace lacks, which reports the prompt and exits with status 3. - A unit test that a prompt reference with no owner and no tag, such as `support-system`, makes the prompt check request `GET /api/v1/commits/-/support-system/latest`, one that `my-org/support-system:prod` requests `GET /api/v1/commits/my-org/support-system/prod`, and one that `-/support-system:prod` requests `GET /api/v1/commits/-/support-system/prod`. - - An integration test that runs verify with `--workspace-id` set to the destination workspace, for an assistant whose referenced prompt tag exists only in that workspace, and asserts that verify exits with status 0 and that `checks` reports the prompt as found. + - An integration test that runs verify with `--workspace-id` set to the destination workspace, for an assistant whose referenced prompt tag exists only in that workspace, and asserts that verify exits with status 0 and that `checks` reports the prompt as found. This test carries `#[ignore = "needs task 2.1"]`, because until then the destination workspace is the test workspace. - A CLI test that `langstar -f json assistant verify` prints one JSON object with `status`, `differences` and `checks` to stdout. - [ ] 9.3 Document assistant compare and verify; verify that each documented command runs as written @@ -157,7 +157,7 @@ Each integration test in this group carries `#[cfg_attr(not(feature = "integrati - An integration test that creates an assistant with a chosen ID on the destination deployment, writes a destination record that holds that ID as pending, imports the file, and asserts that the deployment holds one assistant from the file, with that ID, and that the record marks the ID created. - [ ] 10.2 Add `--update`, `--assistant-id` and `--dry-run`. Verify with these tests: - An integration test that a repeated import creates no assistant. - - An integration test that `--update` changes the recorded assistant, moving it from `test_graph` to `test_graph_alt` among the fields. + - An integration test that `--update` changes the recorded assistant's name, configuration, context and metadata on the existing test deployment, and a second integration test, carrying `#[ignore = "needs task 2.4"]`, that `--update` moves the recorded assistant from `test_graph` to `test_graph_alt` on the destination deployment from 2.4. - An integration test that `--update --assistant-id ` naming an assistant other than the recorded one updates that assistant, leaves the recorded one unchanged and writes the new ID to the destination record, and that `--assistant-id ` without `--update`, against an assistant that differs from the file, creates and changes no assistant and exits with status 3. - An integration test that `--update` of an assistant with metadata `{"kept": 1, "stale": 2}` from a file with `{"kept": 1}` leaves `{"kept": 1}`, or reports `/metadata/stale` and exits with status 3, whichever task 1.2 found the server does. - An integration test that `--update` of an assistant with a description and context from a file with neither clears both, or reports `/description` and `/context` and exits with status 3, whichever task 1.2 found the server does. From 069d0b94038f6be779c31a2b372f924ab0ebf9f1 Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Wed, 28 Oct 2026 12:33:01 -0400 Subject: [PATCH 81/90] =?UTF-8?q?=F0=9F=93=9A=20docs(openspec):=20drop=20t?= =?UTF-8?q?he=20local=20langgraph=20dev=20server=20as=20an=20assistant=20i?= =?UTF-8?q?mport=20target?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Copilot pointed out that --deployment resolves a name or ID only through the control plane's deployment list (resolve_deployment_url in cli/src/deployment_utils.rs), so the proposed commands cannot reach a local langgraph dev server. Nothing else in the change depends on one, so the design drops the claim and lists a server the control plane does not list as a non-goal. Co-Authored-By: Claude Opus 5.5 --- .../changes/export-import-prompts-and-assistants/design.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/openspec/changes/export-import-prompts-and-assistants/design.md b/openspec/changes/export-import-prompts-and-assistants/design.md index 2e485e54a..d52c686a7 100644 --- a/openspec/changes/export-import-prompts-and-assistants/design.md +++ b/openspec/changes/export-import-prompts-and-assistants/design.md @@ -20,6 +20,7 @@ Langstar reads one API key and one workspace at a time, from flags, environment - Copy a prompt or assistant between environments in one command. - Export or import every prompt in a workspace, or every assistant on a deployment, in one run. +- Import onto an Agent Server the control plane does not list, such as a local `langgraph dev` server. The assistant commands find a deployment with `--deployment `, which [`resolve_deployment_url` in cli/src/deployment_utils.rs:23](https://github.com/codekiln/langstar/blob/7850530475f49eb6374652855d6f3a04741899d3/cli/src/deployment_utils.rs#L23) looks up in the control plane's deployment list, and the destination record keys an assistant by that deployment's ID. - Redesign `prompt create`, `get` and `update`, which [#728 Implement CRUD command structure (create, get, update) for prompt commands](https://github.com/codekiln/langstar/issues/728) covers. ## Decisions @@ -150,7 +151,7 @@ The prompt's manifest carries the model and its settings, so the prompt runs wit ### The tests run in the single existing workspace and test deployment -The integration tests import each prompt into the existing test workspace under a fresh handle, and each assistant onto the existing test deployment as a new assistant. That covers import, compare, verify and the destination record, but not a move between workspaces or deployments. The tests that need a second environment carry an `#[ignore]` naming the task that adds it: the assistant graph moves to `test_graph_alt` in [export-import-prompts-and-assistants Task 3.3 Add the assistant fields to the SDK](tasks.md#L29) and [export-import-prompts-and-assistants Task 10.2 Add assistant import --update](tasks.md#L158) need the destination deployment from [export-import-prompts-and-assistants Task 2.4 Add a destination deployment fixture](tasks.md#L19), and the test in [export-import-prompts-and-assistants Task 9.2 Add assistant verify](tasks.md#L138) of a prompt that only the destination workspace holds needs the workspace from [export-import-prompts-and-assistants Task 2.1 Add the destination workspace and the model provider key](tasks.md#L16). Trying export and compare on a real set of assistants and prompts, with langstar's current commands and the SDK, showed that one workspace and one test deployment are enough. Only the assistants needed to move; the prompts stayed in one shared workspace, and the assistants name them without an owner or a commit. Every failure the trial found shows up in an export, or in an import to any deployment, including a local `langgraph dev` server, so a local `langgraph dev` server can serve as an import target. The trial needed no live model call either: comparing each structured prompt's schema and model settings finds a bad copy, and a live call tests the model, not the copy. +The integration tests import each prompt into the existing test workspace under a fresh handle, and each assistant onto the existing test deployment as a new assistant. That covers import, compare, verify and the destination record, but not a move between workspaces or deployments. The tests that need a second environment carry an `#[ignore]` naming the task that adds it: the assistant graph moves to `test_graph_alt` in [export-import-prompts-and-assistants Task 3.3 Add the assistant fields to the SDK](tasks.md#L29) and [export-import-prompts-and-assistants Task 10.2 Add assistant import --update](tasks.md#L158) need the destination deployment from [export-import-prompts-and-assistants Task 2.4 Add a destination deployment fixture](tasks.md#L19), and the test in [export-import-prompts-and-assistants Task 9.2 Add assistant verify](tasks.md#L138) of a prompt that only the destination workspace holds needs the workspace from [export-import-prompts-and-assistants Task 2.1 Add the destination workspace and the model provider key](tasks.md#L16). Trying export and compare on a real set of assistants and prompts, with langstar's current commands and the SDK, showed that one workspace and one test deployment are enough. Only the assistants needed to move; the prompts stayed in one shared workspace, and the assistants name them without an owner or a commit. Every failure the trial found shows up in an export, or in an import to any deployment. The trial needed no live model call either: comparing each structured prompt's schema and model settings finds a bad copy, and a live call tests the model, not the copy. ## Risks / Trade-offs From 4e85e4d81dee41607ef7a3be927480215b13ad8d Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Fri, 30 Oct 2026 09:28:40 -0400 Subject: [PATCH 82/90] =?UTF-8?q?=F0=9F=93=9A=20docs(openspec):=20require?= =?UTF-8?q?=20a=20second=20workspace=20and=20deployment=20before=20the=20c?= =?UTF-8?q?hange=20is=20archived?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit #807 asks the assistant tests to create an assistant on a separate deployment and to exercise an assistant whose prompts were imported into a separate workspace. The design made tasks 2.1, 2.3 and 2.4 optional, so the change could be archived with those tests still ignored. Developers still build and run everything in one workspace and deployment. Tasks 2.1, 2.3 and 2.4 now merge before the archive, each removes the `#[ignore]` from the tests that waited for it, and the archiving pull request links a CI run in which they passed. A 409 from the control plane in 2.3 goes to codekiln as a decision. Co-Authored-By: Claude Opus 5 --- .../changes/export-import-prompts-and-assistants/design.md | 6 ++++-- .../export-import-prompts-and-assistants/proposal.md | 2 +- .../changes/export-import-prompts-and-assistants/tasks.md | 5 +++-- 3 files changed, 8 insertions(+), 5 deletions(-) diff --git a/openspec/changes/export-import-prompts-and-assistants/design.md b/openspec/changes/export-import-prompts-and-assistants/design.md index d52c686a7..2ab02cd22 100644 --- a/openspec/changes/export-import-prompts-and-assistants/design.md +++ b/openspec/changes/export-import-prompts-and-assistants/design.md @@ -149,9 +149,11 @@ Import writes, and `LANGSMITH_WORKSPACE_ID` is easy to leave pointing at the wro The prompt's manifest carries the model and its settings, so the prompt runs without a saved model configuration, and the secret check is the one that finds a prompt that cannot authenticate. Failing on the note would make verify exit with 3 for a prompt that runs. -### The tests run in the single existing workspace and test deployment +### The tests run in one workspace and deployment, and in a second of each before the change is archived -The integration tests import each prompt into the existing test workspace under a fresh handle, and each assistant onto the existing test deployment as a new assistant. That covers import, compare, verify and the destination record, but not a move between workspaces or deployments. The tests that need a second environment carry an `#[ignore]` naming the task that adds it: the assistant graph moves to `test_graph_alt` in [export-import-prompts-and-assistants Task 3.3 Add the assistant fields to the SDK](tasks.md#L29) and [export-import-prompts-and-assistants Task 10.2 Add assistant import --update](tasks.md#L158) need the destination deployment from [export-import-prompts-and-assistants Task 2.4 Add a destination deployment fixture](tasks.md#L19), and the test in [export-import-prompts-and-assistants Task 9.2 Add assistant verify](tasks.md#L138) of a prompt that only the destination workspace holds needs the workspace from [export-import-prompts-and-assistants Task 2.1 Add the destination workspace and the model provider key](tasks.md#L16). Trying export and compare on a real set of assistants and prompts, with langstar's current commands and the SDK, showed that one workspace and one test deployment are enough. Only the assistants needed to move; the prompts stayed in one shared workspace, and the assistants name them without an owner or a commit. Every failure the trial found shows up in an export, or in an import to any deployment. The trial needed no live model call either: comparing each structured prompt's schema and model settings finds a bad copy, and a live call tests the model, not the copy. +The integration tests import each prompt into the existing test workspace under a fresh handle, and each assistant onto the existing test deployment as a new assistant. That covers import, compare, verify and the destination record, but not a move between workspaces or deployments, which [#807 Export, import, compare, and verify prompts and assistants across LangSmith environments](https://github.com/codekiln/langstar/issues/807) asks for: its acceptance criteria have the assistant tests create an assistant on a separate deployment and exercise an assistant whose prompts were imported into a separate workspace. A test that imports into the environment it exported from cannot catch an import that ignores the destination workspace, or a deployment lookup that resolves to the source deployment, because the source and the destination are the same place. The tests that need a second environment carry an `#[ignore]` naming the task that adds it: the assistant graph moves to `test_graph_alt` in [export-import-prompts-and-assistants Task 3.3 Add the assistant fields to the SDK](tasks.md#L29) and [export-import-prompts-and-assistants Task 10.2 Add assistant import --update](tasks.md#L158) need the destination deployment from [export-import-prompts-and-assistants Task 2.4 Add a destination deployment fixture](tasks.md#L19), and the test in [export-import-prompts-and-assistants Task 9.2 Add assistant verify](tasks.md#L138) of a prompt that only the destination workspace holds needs the workspace from [export-import-prompts-and-assistants Task 2.1 Add the destination workspace and the model provider key](tasks.md#L16). Those tasks land before the change is archived, each one's pull request removes the `#[ignore]` from the tests that waited for it, and the pull request that archives the change links a CI run in which those tests passed. The one answer that can change this is a 409 from the control plane in [export-import-prompts-and-assistants Task 2.3 Register the test graph under a second graph ID](tasks.md#L18), which would mean this repository cannot hold a second deployment; the developer reports that answer to codekiln, who decides what the assistant tests run against instead. + +Trying export and compare on a real set of assistants and prompts, with langstar's current commands and the SDK, showed that one workspace and one test deployment are enough for the rest of the work. Only the assistants needed to move; the prompts stayed in one shared workspace, and the assistants name them without an owner or a commit. Every failure the trial found shows up in an export, or in an import to any deployment. The trial needed no live model call either: comparing each structured prompt's schema and model settings finds a bad copy, and a live call tests the model, not the copy. ## Risks / Trade-offs diff --git a/openspec/changes/export-import-prompts-and-assistants/proposal.md b/openspec/changes/export-import-prompts-and-assistants/proposal.md index d22bcb43a..c6ef2feef 100644 --- a/openspec/changes/export-import-prompts-and-assistants/proposal.md +++ b/openspec/changes/export-import-prompts-and-assistants/proposal.md @@ -33,7 +33,7 @@ A prompt belongs to a workspace and an assistant belongs to a deployment, but bo - SDK: `sdk/src/prompts.rs` gains the prompt calls export and import need: a commit read that asks for the model and returns it on `CommitManifestResponse`, a commit create that sends a description and returns the new commit's `id`, a commit list, calls that list, create, move and delete commit tags, and a repo update. `Prompt` gains the repo's readme, tags and `tenant_id`, and export fills `source.workspace_id` from `tenant_id`. `sdk/src/assistants.rs` gains the `context` and `version` fields, `assistant_id` and `if_exists` on create, `graph_id` on update, and a search by graph ID. Developers commit these fields as `✨ feat(sdk)` with no `BREAKING CHANGE:` footer, so the Prepare Release workflow makes the next release a minor version. [design.md's decision on the SDK release](design.md#developers-commit-the-new-sdk-fields-as--featsdk-so-the-next-release-is-a-minor-version) links the release docs that ask for this. - CLI: new subcommands in `cli/src/commands/prompt.rs` and `cli/src/commands/assistant.rs`, and a shared module for the export file, the destination record, comparison and the exit status. -- Tests that copy between environments need a second workspace, a second deployment and one model provider key for the structured output check, and the structured output check skips without the key. The tests run in the single existing workspace and test deployment until trying export and import on real assistants and prompts shows whether they need a second workspace and deployment. Once they exist, langstar's tests read them through `fnox.toml` and the CI workflow; the destination deployment fixture from [export-import-prompts-and-assistants Task 2.4 Add a destination deployment fixture](tasks.md#L19) creates the deployment whenever the cleanup workflow has deleted it, as the source fixture does today. Nobody knows yet whether the control plane accepts a second deployment from this repository. In [export-import-prompts-and-assistants Task 2.3 Register the test graph under a second graph ID](tasks.md#L18), a developer deploys the new `langgraph.json` once to find out, before anyone builds the fixture. +- Tests that copy between environments need a second workspace, a second deployment and one model provider key for the structured output check, and the structured output check skips without the key. The tests run in the single existing workspace and test deployment while developers build the commands, and in a second workspace and a second deployment before the change is archived, because [#807 Export, import, compare, and verify prompts and assistants across LangSmith environments](https://github.com/codekiln/langstar/issues/807) asks for a copy between environments. Once the second workspace and deployment exist, langstar's tests read them through `fnox.toml` and the CI workflow; the destination deployment fixture from [export-import-prompts-and-assistants Task 2.4 Add a destination deployment fixture](tasks.md#L19) creates the deployment whenever the cleanup workflow has deleted it, as the source fixture does today. Nobody knows yet whether the control plane accepts a second deployment from this repository. In [export-import-prompts-and-assistants Task 2.3 Register the test graph under a second graph ID](tasks.md#L18), a developer deploys the new `langgraph.json` once to find out, before anyone builds the fixture. - Docs: a new usage page on moving prompts and assistants between environments. ## Citations diff --git a/openspec/changes/export-import-prompts-and-assistants/tasks.md b/openspec/changes/export-import-prompts-and-assistants/tasks.md index 75c57ef6b..d4fa06ba7 100644 --- a/openspec/changes/export-import-prompts-and-assistants/tasks.md +++ b/openspec/changes/export-import-prompts-and-assistants/tasks.md @@ -1,6 +1,6 @@ # Tasks -Each numbered group below is one pull request, merged in the order listed, except group 2, whose tasks each get a pull request of their own. Tasks 2.3 and 2.4 merge in that order: the fixture in 2.4 deploys from `main`, so the `langgraph.json` from 2.3 has to be on `main` first. Task 2.2 can merge at any time. Tasks 2.1, 2.3 and 2.4 apply only if the tests need a second workspace and deployment, as `design.md`'s decision on test environments says. Until they merge, a task below that names the destination workspace means the existing test workspace from `LANGSMITH_WORKSPACE_ID`, into which each test imports a prompt under a fresh handle, and a task that names the destination deployment means the existing test deployment. The tests that need a second environment carry `#[ignore = "needs task 2.1"]` or `#[ignore = "needs task 2.4"]`, and the pull request for that task removes the attribute: the graph moves in 3.3 and 10.2 need `test_graph_alt` on the destination deployment from 2.4, and the prompt in 9.2 that exists only in the destination workspace needs the workspace from 2.1. The structured output check in 7.3 skips without an OpenAI key. +Each numbered group below is one pull request, merged in the order listed, except group 2, whose tasks each get a pull request of their own. Tasks 2.3 and 2.4 merge in that order: the fixture in 2.4 deploys from `main`, so the `langgraph.json` from 2.3 has to be on `main` first. Task 2.2 can merge at any time. Tasks 2.1, 2.3 and 2.4 add the second workspace and deployment, and they merge before the change is archived, as `design.md`'s decision on test environments says. Every other group can be built and merged before them. Until they merge, a task below that names the destination workspace means the existing test workspace from `LANGSMITH_WORKSPACE_ID`, into which each test imports a prompt under a fresh handle, and a task that names the destination deployment means the existing test deployment. The tests that need a second environment carry `#[ignore = "needs task 2.1"]` or `#[ignore = "needs task 2.4"]`, and the pull request for that task removes the attribute: the graph moves in 3.3 and 10.2 need `test_graph_alt` on the destination deployment from 2.4, and the prompt in 9.2 that exists only in the destination workspace needs the workspace from 2.1. The structured output check in 7.3 skips without an OpenAI key. ## 1. Check the API round trip before building on it @@ -13,7 +13,7 @@ Each numbered group below is one pull request, merged in the order listed, excep ## 2. Test environments -- [ ] 2.1 Add the destination workspace and the model provider key to `fnox.toml`, the CI workflow and `docs/dev/environment-variables.md`, with placeholders only; verify with `mise run check` and a CI run that reads both. This task applies only if the tests need a second workspace, and adds the key only once CI has one. The import tests need the workspace as a place to import into, and its secret lets prompt import's secret check pass for a prompt with an OpenAI model. The key lets the check in 7.3 run each imported structured prompt once against a small OpenAI model, which bills that account on every CI run; it is the one test that shows a copy still returns structured data, where every other test compares text. +- [ ] 2.1 Add the destination workspace and the model provider key to `fnox.toml`, the CI workflow and `docs/dev/environment-variables.md`, with placeholders only; verify with `mise run check` and a CI run that reads both. This task adds the model provider key only once CI has one. The import tests need the workspace as a place to import into, and its secret lets prompt import's secret check pass for a prompt with an OpenAI model. The key lets the check in 7.3 run each imported structured prompt once against a small OpenAI model, which bills that account on every CI run; it is the one test that shows a copy still returns structured data, where every other test compares text. - [ ] 2.2 Replace the bare `#[ignore]` on `test_assistant_create_basic` in `cli/tests/assistant_command_test.rs`, whose reason says a deployment built from GitHub has no `custom_url`, with `#[cfg_attr(not(feature = "integration-tests"), ignore)]`, the guard `test_assistant_description_field` beside it carries. The control plane sets `source_config.custom_url` on a deployment built from GitHub, as `SourceConfig.custom_url` in `reference/openapi/langchain/control-plane/openapi.json` says. Verify by running the test against the test deployment. - [ ] 2.3 Add a second `langgraph.json` under `tests/fixtures/` that registers the test graph from `tests/fixtures/test-graph-deployment/langgraph.json` under two graph IDs, `test_graph` and `test_graph_alt`, so the update tests in 3.3 and 10.2 can move an assistant from one graph to the other. To verify, deploy the new file while the source test deployment is live, which shows whether the control plane accepts a second deployment from this repository: run `langstar deployment create` with `--branch` set to this pull request's branch and `--config-path` set to the new file. A 409 "A deployment already exists for this agent environment" means it does not, and the developer reports that to codekiln before anyone starts task 2.4. Otherwise run `langstar assistant list --deployment `, find one assistant with graph ID `test_graph` and one with `test_graph_alt`, then delete the deployment. - [ ] 2.4 Add a destination deployment fixture to `sdk/src/test_utils.rs` and `cli/tests/common/fixtures.rs` that deploys the `langgraph.json` from 2.3 under the name prefix `dest-integration-test`. `find_reusable_deployment` returns any deployment whose name starts with the fixture's prefix before it checks the graph config, so this prefix keeps the source fixture, whose prefix is `pr-integration-test`, from picking up the destination deployment. `find_reusable_deployment` then falls back to any live deployment built from the same repository and graph config file, so the separate file from 2.3 keeps the destination fixture from picking up the source deployment. The control plane answers a second deployment of one agent environment with 409 "A deployment already exists for this agent environment". No document says which fields make an agent environment, and the verification in 2.3 shows whether a second graph config file in the same repository makes a new one. `.github/workflows/cleanup-test-deployments.yml` runs every four hours and deletes each deployment whose name contains `integration-test` once it is four hours old or older, so the destination deployment is deleted and rebuilt on the same schedule as the source deployment. Verify with an integration test that asserts the destination and source deployment IDs differ and lists assistants on the destination deployment with `langstar assistant list` @@ -168,4 +168,5 @@ Each integration test in this group carries `#[cfg_attr(not(feature = "integrati ## Workflow follow-up +- Archive the change only once 2.1, 2.3 and 2.4 have merged and a CI run shows the tests they unignore passing, or codekiln has decided what those tests run against after a 409 in 2.3. - Run `openspec validate export-import-prompts-and-assistants --strict`, then archive the change in the pull request for group 10. From a2c587d45e677c160b0c50426c81d95844d82445 Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Fri, 30 Oct 2026 09:33:03 -0400 Subject: [PATCH 83/90] =?UTF-8?q?=F0=9F=93=9A=20docs(openspec):=20give=20t?= =?UTF-8?q?he=20test-environment=20decision=20subjects=20that=20can=20act?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The pr-language-review skill found documents and tasks doing the verbs: "the design made tasks optional", "those tasks land", "they merge", "a 409 goes to codekiln". Developers merge the tasks, the control plane answers with a 409, and codekiln decides. Also drops the coined "unignore" from the Workflow follow-up and splits the 409 clause out of the sentence it was knotted into. Co-Authored-By: Claude Opus 5 --- .../changes/export-import-prompts-and-assistants/design.md | 4 ++-- .../changes/export-import-prompts-and-assistants/proposal.md | 2 +- .../changes/export-import-prompts-and-assistants/tasks.md | 4 ++-- 3 files changed, 5 insertions(+), 5 deletions(-) diff --git a/openspec/changes/export-import-prompts-and-assistants/design.md b/openspec/changes/export-import-prompts-and-assistants/design.md index 2ab02cd22..5a67fed4b 100644 --- a/openspec/changes/export-import-prompts-and-assistants/design.md +++ b/openspec/changes/export-import-prompts-and-assistants/design.md @@ -149,9 +149,9 @@ Import writes, and `LANGSMITH_WORKSPACE_ID` is easy to leave pointing at the wro The prompt's manifest carries the model and its settings, so the prompt runs without a saved model configuration, and the secret check is the one that finds a prompt that cannot authenticate. Failing on the note would make verify exit with 3 for a prompt that runs. -### The tests run in one workspace and deployment, and in a second of each before the change is archived +### Developers run the tests in one workspace and deployment, and in a second workspace and a second deployment before they archive the change -The integration tests import each prompt into the existing test workspace under a fresh handle, and each assistant onto the existing test deployment as a new assistant. That covers import, compare, verify and the destination record, but not a move between workspaces or deployments, which [#807 Export, import, compare, and verify prompts and assistants across LangSmith environments](https://github.com/codekiln/langstar/issues/807) asks for: its acceptance criteria have the assistant tests create an assistant on a separate deployment and exercise an assistant whose prompts were imported into a separate workspace. A test that imports into the environment it exported from cannot catch an import that ignores the destination workspace, or a deployment lookup that resolves to the source deployment, because the source and the destination are the same place. The tests that need a second environment carry an `#[ignore]` naming the task that adds it: the assistant graph moves to `test_graph_alt` in [export-import-prompts-and-assistants Task 3.3 Add the assistant fields to the SDK](tasks.md#L29) and [export-import-prompts-and-assistants Task 10.2 Add assistant import --update](tasks.md#L158) need the destination deployment from [export-import-prompts-and-assistants Task 2.4 Add a destination deployment fixture](tasks.md#L19), and the test in [export-import-prompts-and-assistants Task 9.2 Add assistant verify](tasks.md#L138) of a prompt that only the destination workspace holds needs the workspace from [export-import-prompts-and-assistants Task 2.1 Add the destination workspace and the model provider key](tasks.md#L16). Those tasks land before the change is archived, each one's pull request removes the `#[ignore]` from the tests that waited for it, and the pull request that archives the change links a CI run in which those tests passed. The one answer that can change this is a 409 from the control plane in [export-import-prompts-and-assistants Task 2.3 Register the test graph under a second graph ID](tasks.md#L18), which would mean this repository cannot hold a second deployment; the developer reports that answer to codekiln, who decides what the assistant tests run against instead. +The integration tests import each prompt into the existing test workspace under a fresh handle, and each assistant onto the existing test deployment as a new assistant. That covers import, compare, verify and the destination record, but not a move between workspaces or deployments, which [#807 Export, import, compare, and verify prompts and assistants across LangSmith environments](https://github.com/codekiln/langstar/issues/807) asks for. Its acceptance criteria ask a developer to write assistant tests that create an assistant on a separate deployment, and to run an assistant whose prompts another developer imported into a separate workspace. When the source and the destination are one workspace and one deployment, an import that ignores `--workspace-id`, or that writes to the deployment it read from, passes every test anyway, because it writes where the test looks. The tests that need a second environment carry an `#[ignore]` naming the task that adds it: the assistant graph moves to `test_graph_alt` in [export-import-prompts-and-assistants Task 3.3 Add the assistant fields to the SDK](tasks.md#L29) and [export-import-prompts-and-assistants Task 10.2 Add assistant import --update](tasks.md#L158) need the destination deployment from [export-import-prompts-and-assistants Task 2.4 Add a destination deployment fixture](tasks.md#L19), and the test in [export-import-prompts-and-assistants Task 9.2 Add assistant verify](tasks.md#L138) of a prompt that only the destination workspace holds needs the workspace from [export-import-prompts-and-assistants Task 2.1 Add the destination workspace and the model provider key](tasks.md#L16). Developers merge tasks 2.1, 2.3 and 2.4 before anyone archives the change. The pull request for each one removes the `#[ignore]` from the tests that waited for it, and the pull request that archives the change links a CI run in which those tests passed. If the control plane answers [export-import-prompts-and-assistants Task 2.3 Register the test graph under a second graph ID](tasks.md#L18) with a 409, this repository cannot hold a second deployment, and the developer reports that answer to codekiln, who decides what the assistant tests run against. Trying export and compare on a real set of assistants and prompts, with langstar's current commands and the SDK, showed that one workspace and one test deployment are enough for the rest of the work. Only the assistants needed to move; the prompts stayed in one shared workspace, and the assistants name them without an owner or a commit. Every failure the trial found shows up in an export, or in an import to any deployment. The trial needed no live model call either: comparing each structured prompt's schema and model settings finds a bad copy, and a live call tests the model, not the copy. diff --git a/openspec/changes/export-import-prompts-and-assistants/proposal.md b/openspec/changes/export-import-prompts-and-assistants/proposal.md index c6ef2feef..a1b2e5561 100644 --- a/openspec/changes/export-import-prompts-and-assistants/proposal.md +++ b/openspec/changes/export-import-prompts-and-assistants/proposal.md @@ -33,7 +33,7 @@ A prompt belongs to a workspace and an assistant belongs to a deployment, but bo - SDK: `sdk/src/prompts.rs` gains the prompt calls export and import need: a commit read that asks for the model and returns it on `CommitManifestResponse`, a commit create that sends a description and returns the new commit's `id`, a commit list, calls that list, create, move and delete commit tags, and a repo update. `Prompt` gains the repo's readme, tags and `tenant_id`, and export fills `source.workspace_id` from `tenant_id`. `sdk/src/assistants.rs` gains the `context` and `version` fields, `assistant_id` and `if_exists` on create, `graph_id` on update, and a search by graph ID. Developers commit these fields as `✨ feat(sdk)` with no `BREAKING CHANGE:` footer, so the Prepare Release workflow makes the next release a minor version. [design.md's decision on the SDK release](design.md#developers-commit-the-new-sdk-fields-as--featsdk-so-the-next-release-is-a-minor-version) links the release docs that ask for this. - CLI: new subcommands in `cli/src/commands/prompt.rs` and `cli/src/commands/assistant.rs`, and a shared module for the export file, the destination record, comparison and the exit status. -- Tests that copy between environments need a second workspace, a second deployment and one model provider key for the structured output check, and the structured output check skips without the key. The tests run in the single existing workspace and test deployment while developers build the commands, and in a second workspace and a second deployment before the change is archived, because [#807 Export, import, compare, and verify prompts and assistants across LangSmith environments](https://github.com/codekiln/langstar/issues/807) asks for a copy between environments. Once the second workspace and deployment exist, langstar's tests read them through `fnox.toml` and the CI workflow; the destination deployment fixture from [export-import-prompts-and-assistants Task 2.4 Add a destination deployment fixture](tasks.md#L19) creates the deployment whenever the cleanup workflow has deleted it, as the source fixture does today. Nobody knows yet whether the control plane accepts a second deployment from this repository. In [export-import-prompts-and-assistants Task 2.3 Register the test graph under a second graph ID](tasks.md#L18), a developer deploys the new `langgraph.json` once to find out, before anyone builds the fixture. +- Tests that copy between environments need a second workspace, a second deployment and one model provider key for the structured output check, and the structured output check skips without the key. Developers run the tests in the single existing workspace and test deployment while they build the commands, and in a second workspace and a second deployment before anyone archives the change, because [#807 Export, import, compare, and verify prompts and assistants across LangSmith environments](https://github.com/codekiln/langstar/issues/807) asks for a copy between environments. Once the second workspace and deployment exist, langstar's tests read them through `fnox.toml` and the CI workflow; the destination deployment fixture from [export-import-prompts-and-assistants Task 2.4 Add a destination deployment fixture](tasks.md#L19) creates the deployment whenever the cleanup workflow has deleted it, as the source fixture does today. Nobody knows yet whether the control plane accepts a second deployment from this repository. In [export-import-prompts-and-assistants Task 2.3 Register the test graph under a second graph ID](tasks.md#L18), a developer deploys the new `langgraph.json` once to find out, before anyone builds the fixture. - Docs: a new usage page on moving prompts and assistants between environments. ## Citations diff --git a/openspec/changes/export-import-prompts-and-assistants/tasks.md b/openspec/changes/export-import-prompts-and-assistants/tasks.md index d4fa06ba7..27f1af996 100644 --- a/openspec/changes/export-import-prompts-and-assistants/tasks.md +++ b/openspec/changes/export-import-prompts-and-assistants/tasks.md @@ -1,6 +1,6 @@ # Tasks -Each numbered group below is one pull request, merged in the order listed, except group 2, whose tasks each get a pull request of their own. Tasks 2.3 and 2.4 merge in that order: the fixture in 2.4 deploys from `main`, so the `langgraph.json` from 2.3 has to be on `main` first. Task 2.2 can merge at any time. Tasks 2.1, 2.3 and 2.4 add the second workspace and deployment, and they merge before the change is archived, as `design.md`'s decision on test environments says. Every other group can be built and merged before them. Until they merge, a task below that names the destination workspace means the existing test workspace from `LANGSMITH_WORKSPACE_ID`, into which each test imports a prompt under a fresh handle, and a task that names the destination deployment means the existing test deployment. The tests that need a second environment carry `#[ignore = "needs task 2.1"]` or `#[ignore = "needs task 2.4"]`, and the pull request for that task removes the attribute: the graph moves in 3.3 and 10.2 need `test_graph_alt` on the destination deployment from 2.4, and the prompt in 9.2 that exists only in the destination workspace needs the workspace from 2.1. The structured output check in 7.3 skips without an OpenAI key. +Each numbered group below is one pull request, merged in the order listed, except group 2, whose tasks each get a pull request of their own. Tasks 2.3 and 2.4 merge in that order: the fixture in 2.4 deploys from `main`, so the `langgraph.json` from 2.3 has to be on `main` first. Task 2.2 can merge at any time. Tasks 2.1, 2.3 and 2.4 add the second workspace and deployment, and developers merge them before anyone archives the change, as `design.md`'s decision on test environments says. Developers can build and merge every other group before them. Until they merge, a task below that names the destination workspace means the existing test workspace from `LANGSMITH_WORKSPACE_ID`, into which each test imports a prompt under a fresh handle, and a task that names the destination deployment means the existing test deployment. The tests that need a second environment carry `#[ignore = "needs task 2.1"]` or `#[ignore = "needs task 2.4"]`, and the pull request for that task removes the attribute: the graph moves in 3.3 and 10.2 need `test_graph_alt` on the destination deployment from 2.4, and the prompt in 9.2 that exists only in the destination workspace needs the workspace from 2.1. The structured output check in 7.3 skips without an OpenAI key. ## 1. Check the API round trip before building on it @@ -168,5 +168,5 @@ Each integration test in this group carries `#[cfg_attr(not(feature = "integrati ## Workflow follow-up -- Archive the change only once 2.1, 2.3 and 2.4 have merged and a CI run shows the tests they unignore passing, or codekiln has decided what those tests run against after a 409 in 2.3. +- Archive the change once 2.1, 2.3 and 2.4 have merged and a CI run shows the tests whose `#[ignore]` those tasks removed passing. If the control plane answered 2.3 with a 409, archive it once codekiln has decided what the assistant tests run against. - Run `openspec validate export-import-prompts-and-assistants --strict`, then archive the change in the pull request for group 10. From 7d4841d776e30b449f77f22dfd497ee86e7c4ce9 Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Fri, 30 Oct 2026 09:34:52 -0400 Subject: [PATCH 84/90] =?UTF-8?q?=F0=9F=93=9A=20docs(openspec):=20name=20t?= =?UTF-8?q?he=20tasks=20the=20archive=20waits=20on,=20and=20shorten=20the?= =?UTF-8?q?=20heading?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The decision said "tasks 2.1, 2.3 and 2.4" with no link and no title, which costs a reader a trip into tasks.md to learn what they add. Each one now carries the link and the task title the rest of design.md uses. The heading was a sentence long enough to read twice. It now says what developers do: add the second workspace and deployment before they archive the change. Co-Authored-By: Claude Opus 5 --- .../changes/export-import-prompts-and-assistants/design.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/openspec/changes/export-import-prompts-and-assistants/design.md b/openspec/changes/export-import-prompts-and-assistants/design.md index 5a67fed4b..db52a832e 100644 --- a/openspec/changes/export-import-prompts-and-assistants/design.md +++ b/openspec/changes/export-import-prompts-and-assistants/design.md @@ -149,9 +149,9 @@ Import writes, and `LANGSMITH_WORKSPACE_ID` is easy to leave pointing at the wro The prompt's manifest carries the model and its settings, so the prompt runs without a saved model configuration, and the secret check is the one that finds a prompt that cannot authenticate. Failing on the note would make verify exit with 3 for a prompt that runs. -### Developers run the tests in one workspace and deployment, and in a second workspace and a second deployment before they archive the change +### Developers add the second workspace and deployment before they archive the change -The integration tests import each prompt into the existing test workspace under a fresh handle, and each assistant onto the existing test deployment as a new assistant. That covers import, compare, verify and the destination record, but not a move between workspaces or deployments, which [#807 Export, import, compare, and verify prompts and assistants across LangSmith environments](https://github.com/codekiln/langstar/issues/807) asks for. Its acceptance criteria ask a developer to write assistant tests that create an assistant on a separate deployment, and to run an assistant whose prompts another developer imported into a separate workspace. When the source and the destination are one workspace and one deployment, an import that ignores `--workspace-id`, or that writes to the deployment it read from, passes every test anyway, because it writes where the test looks. The tests that need a second environment carry an `#[ignore]` naming the task that adds it: the assistant graph moves to `test_graph_alt` in [export-import-prompts-and-assistants Task 3.3 Add the assistant fields to the SDK](tasks.md#L29) and [export-import-prompts-and-assistants Task 10.2 Add assistant import --update](tasks.md#L158) need the destination deployment from [export-import-prompts-and-assistants Task 2.4 Add a destination deployment fixture](tasks.md#L19), and the test in [export-import-prompts-and-assistants Task 9.2 Add assistant verify](tasks.md#L138) of a prompt that only the destination workspace holds needs the workspace from [export-import-prompts-and-assistants Task 2.1 Add the destination workspace and the model provider key](tasks.md#L16). Developers merge tasks 2.1, 2.3 and 2.4 before anyone archives the change. The pull request for each one removes the `#[ignore]` from the tests that waited for it, and the pull request that archives the change links a CI run in which those tests passed. If the control plane answers [export-import-prompts-and-assistants Task 2.3 Register the test graph under a second graph ID](tasks.md#L18) with a 409, this repository cannot hold a second deployment, and the developer reports that answer to codekiln, who decides what the assistant tests run against. +The integration tests import each prompt into the existing test workspace under a fresh handle, and each assistant onto the existing test deployment as a new assistant. That covers import, compare, verify and the destination record, but not a move between workspaces or deployments. [#807 Export, import, compare, and verify prompts and assistants across LangSmith environments](https://github.com/codekiln/langstar/issues/807) asks for that move: assistant tests that create an assistant on a separate deployment, and an assistant whose prompts a developer imported into a separate workspace. When the source and the destination are one workspace and one deployment, an import that ignores `--workspace-id`, or that writes to the deployment it read from, passes every test anyway, because it writes where the test looks. The tests that need a second environment carry an `#[ignore]` naming the task that adds it: the assistant graph moves to `test_graph_alt` in [export-import-prompts-and-assistants Task 3.3 Add the assistant fields to the SDK](tasks.md#L29) and [export-import-prompts-and-assistants Task 10.2 Add assistant import --update](tasks.md#L158) need the destination deployment from [export-import-prompts-and-assistants Task 2.4 Add a destination deployment fixture](tasks.md#L19), and the test in [export-import-prompts-and-assistants Task 9.2 Add assistant verify](tasks.md#L138) of a prompt that only the destination workspace holds needs the workspace from [export-import-prompts-and-assistants Task 2.1 Add the destination workspace and the model provider key](tasks.md#L16). Developers merge [export-import-prompts-and-assistants Task 2.1 Add the destination workspace and the model provider key](tasks.md#L16), [export-import-prompts-and-assistants Task 2.3 Register the test graph under a second graph ID](tasks.md#L18) and [export-import-prompts-and-assistants Task 2.4 Add a destination deployment fixture](tasks.md#L19) before anyone archives the change. The pull request for each one removes the `#[ignore]` from the tests that waited for it, and the pull request that archives the change links a CI run in which those tests passed. If the control plane answers [export-import-prompts-and-assistants Task 2.3 Register the test graph under a second graph ID](tasks.md#L18) with a 409, this repository cannot hold a second deployment, and the developer reports that answer to codekiln, who decides what the assistant tests run against. Trying export and compare on a real set of assistants and prompts, with langstar's current commands and the SDK, showed that one workspace and one test deployment are enough for the rest of the work. Only the assistants needed to move; the prompts stayed in one shared workspace, and the assistants name them without an owner or a commit. Every failure the trial found shows up in an export, or in an import to any deployment. The trial needed no live model call either: comparing each structured prompt's schema and model settings finds a bad copy, and a live call tests the model, not the copy. From f8f53b51aaafa0cd572fff7cb5c556e749ad116e Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Fri, 30 Oct 2026 09:37:47 -0400 Subject: [PATCH 85/90] =?UTF-8?q?=F0=9F=93=9A=20docs(openspec):=20rewrite?= =?UTF-8?q?=20the=20verify=20and=20import-check=20sections=20for=20a=20rea?= =?UTF-8?q?der=20who=20has=20not=20read=20the=20LangSmith=20API?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit codekiln read three paragraphs and could not tell what they were saying. The verify section now opens with the question a user has after an import — will the copy run here — and then says what verify asks the destination and what it prints. Each API call arrives after the sentence that says why langstar makes it. The assistant graph check explains that every graph a deployment serves has a default assistant, so an empty search answer means the deployment does not serve the graph; the 422 paragraph says the server may refuse an unknown graph ID instead of answering with nothing. The import section now says what import does — run verify's checks before it creates anything — and gives the cleanup it saves afterwards. Co-Authored-By: Claude Opus 5 --- .../design.md | 24 +++++++++++++------ 1 file changed, 17 insertions(+), 7 deletions(-) diff --git a/openspec/changes/export-import-prompts-and-assistants/design.md b/openspec/changes/export-import-prompts-and-assistants/design.md index db52a832e..7fabc3a62 100644 --- a/openspec/changes/export-import-prompts-and-assistants/design.md +++ b/openspec/changes/export-import-prompts-and-assistants/design.md @@ -99,19 +99,29 @@ Without `--update`, import lists the differences and exits with status 3. `--dry Compare reads the prompt or assistant from the destination, puts it in the shape of the export file's `repo`, `commit` and `assistant` objects, and lists every value that differs from the file. Compare checks every key in those objects, including the chat model's settings and the `id` arrays inside a manifest that name the prompt and model classes, such as `ChatOpenAI`. It skips `commit.model_config` and `commit.model_provider`, because LangSmith fills both in itself when import pushes a commit, and a destination workspace can fill them in differently from the source. Export leaves the values the server sets, such as IDs, version numbers, timestamps, owner and like counts, out of those objects, so they never show up as differences. The source environment's workspace, deployment and IDs sit in `source`, so they never show up as differences against a copy. Each difference names its JSON Pointer path and both values. Before the report prints either side of a difference, it replaces with `` every string that export's credential check flags, by either rule, whether the difference is at that string or at an object that contains it. A credential someone pasted into the destination prompt or the destination assistant, or into the file, therefore stays out of the terminal and the CI log. The report prints a LangChain secret reference, such as `{"lc": 1, "type": "secret", "id": ["OPENAI_API_KEY"]}`, unchanged, because the reference holds only the secret's name, never its value. The report prints to a terminal or a CI log, where the user needs to see where each difference is, so it redacts the string and goes on reporting. For a prompt, the comparison covers the commit description, the commit tags and the commit each one points at, any tag on the compared commit that the file lacks, and the repo's description, readme, tags and visibility as well as the manifest. Import runs the same comparison after it writes, so a field LangSmith or the Agent Server drops shows up as a difference. -### `prompt verify` and `assistant verify` check the destination's secrets, graph and prompts +### `prompt verify` and `assistant verify` answer whether the copy can run -Prompt verify checks that the workspace holds every secret the prompt's model reads. Export lists those names in `requires.secrets` by walking the manifest for LangChain secret references, `{"lc": 1, "type": "secret", "id": ["OPENAI_API_KEY"]}`, the shape the Python SDK's `_manifest_has_secrets` matches. Verify looks each name up among the workspace's secrets (`GET /api/v1/workspaces/current/secrets`, which `langstar secrets list` already calls) and prints the name of each missing secret and no secret value. +A user who has just imported a prompt or an assistant wants one question answered: will it run here? A create request that came back 200 does not answer it. The copy runs only if the destination workspace holds the API keys the prompt's model reads, the destination deployment serves the graph the assistant names, and the destination workspace holds the prompts the assistant refers to. Verify asks the destination about each of those and names what is missing, so the user fixes it before a learner or a test hits the copy. -Prompt verify also notes when the workspace has no saved model configuration for the prompt's model, and leaves the exit status alone, because the prompt runs with the model its manifest carries. Export takes the model's class name, such as `ChatOpenAI`, from the last element of the chat model constructor's `id` in the manifest, such as `["langchain", "chat_models", "openai", "ChatOpenAI"]`, and stores it in `requires.model_class`. Verify looks for a saved model configuration (`GET /api/v1/playground-settings`, which `langstar model-config list` calls) whose settings `id` ends in that class name. A prompt saved without a model has no chat model constructor in its manifest, so export writes `requires.model_class` as null, and verify and import skip this check for it and still run the secret check. [Prompt verify reports a missing saved model configuration as a note, not a failure](#prompt-verify-reports-a-missing-saved-model-configuration-as-a-note-not-a-failure) gives the reason. +Prompt verify starts with the workspace's secrets. A prompt's model needs an API key, and the manifest names the key rather than carrying its value, as `{"lc": 1, "type": "secret", "id": ["OPENAI_API_KEY"]}`; that is the shape the Python SDK's `_manifest_has_secrets` matches. Export walks the manifest for those references and writes each name into `requires.secrets`. Verify reads the workspace's secrets (`GET /api/v1/workspaces/current/secrets`, the request `langstar secrets list` already sends) and prints the name of every one the workspace lacks. It prints names, never values. -Import and verify work out `requires` again from the file's manifest: the secret names, and the model class or null. When either differs from what the file holds, they write nothing, name the field and exit with status 1. This catches a manifest someone edited in Git without updating `requires`. +Prompt verify then looks for a saved model configuration for the prompt's model, and reports a missing one as a note. The prompt still runs without one, because its manifest carries the model and its settings. Export reads the model's class name, such as `ChatOpenAI`, from the last element of the chat model constructor's `id` in the manifest, such as `["langchain", "chat_models", "openai", "ChatOpenAI"]`, and writes it to `requires.model_class`. Verify then looks for a saved model configuration whose settings `id` ends in that class name (`GET /api/v1/playground-settings`, the request `langstar model-config list` sends). A prompt someone saved without a model has no chat model constructor, so export writes `requires.model_class` as null, and verify and import skip this check and still check the secrets. [Prompt verify reports a missing saved model configuration as a note, not a failure](#prompt-verify-reports-a-missing-saved-model-configuration-as-a-note-not-a-failure) gives the reason. -Assistant verify checks the graph with `POST /assistants/search` filtered to the graph ID; every graph on a deployment has a default assistant, so an empty result means the deployment does not serve the graph. The Agent Server's OpenAPI spec lists the deployment's graph IDs as the allowed values of the search's `graph_id`, so the server may answer an unknown graph ID with 422 instead of an empty result. Verify therefore also counts a 422 whose error names `graph_id` as a missing graph, and exits with status 1 on any other error; in [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8) a developer records which answer the live server gives. It checks each prompt in `requires.prompts`, such as `support-system:prod`, by asking LangSmith for the commit that tag points at (`GET /api/v1/commits/{owner}/{repo}/{tag}`, the request `langstar prompt pull --commit prod` already sends). A prompt name has the form `[/][:]`: without an owner verify uses `-`, the current workspace, and without a tag it asks for `latest`. A 404 means the workspace lacks the prompt or its tag. +Import and verify both read the file's manifest and work out `requires` from it again, then compare that with the `requires` the file carries. When the two disagree, they write nothing, name the field and exit with status 1. This catches a manifest someone edited in Git and left `requires` behind. -### A failed dependency check stops import before it writes anything +Assistant verify asks the destination deployment whether it serves the graph the assistant names. The Agent Server has no call that lists a deployment's graphs, so verify searches the deployment's assistants for that graph ID (`POST /assistants/search` with `graph_id`). Every graph a deployment serves has a default assistant, so a deployment that serves the graph answers with at least that one, and an empty answer means the deployment does not serve it. Verify then names the graph ID as missing and the user picks a deployment that serves it. -A copy that cannot run is worse than no copy, because the user then has to find and delete it by hand; asking for the destination's secrets and prompts first costs only the order of the steps. Before it writes anything, import checks that the destination has the secrets, the graph and the prompts the copy needs. These are the checks `verify` makes, and import runs them because [#807 Export, import, compare, and verify prompts and assistants across LangSmith environments](https://github.com/codekiln/langstar/issues/807) asks import to check the destination's dependencies. Prompt import stops when the destination workspace lacks a secret in `requires.secrets`, and assistant import stops when the deployment does not serve the file's graph or the workspace, from `--workspace-id` or the configured workspace, lacks a prompt in `requires.prompts`. Each names what is missing, writes nothing and exits with status 3, and `--dry-run` reports the same failed checks. A user therefore creates the destination workspace's secrets before importing a prompt, and imports the prompts before importing an assistant that refers to them. +The Agent Server may refuse the search instead of answering it with nothing: its OpenAPI spec lists the deployment's graph IDs as the allowed values of `graph_id`, so an unknown graph ID may come back as a 422. Verify reads a 422 whose error names `graph_id` the same way it reads an empty answer, as a graph this deployment does not serve, and exits with status 1 on every other error. In [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8), a developer runs the search against the live server and records which of the two answers it gives. + +Assistant verify also checks each prompt the assistant refers to. `requires.prompts` holds names such as `support-system:prod`, and verify asks LangSmith for the commit each tag points at (`GET /api/v1/commits/{owner}/{repo}/{tag}`, the request `langstar prompt pull --commit prod` already sends). A name has the form `[/][:]`: with no owner, verify uses `-`, which LangSmith reads as the current workspace, and with no tag it asks for `latest`. A 404 means the workspace lacks that prompt or that tag, and verify prints the name. + +### Import runs verify's checks before it writes anything + +Import runs the checks `verify` runs, and it runs them before it creates anything. Prompt import stops when the destination workspace lacks a secret from `requires.secrets`. Assistant import stops when the deployment does not serve the file's graph, or when the workspace — the one from `--workspace-id`, or the configured one — lacks a prompt from `requires.prompts`. Either one names what is missing, writes nothing and exits with status 3, and `--dry-run` reports the same failures. + +Checking first saves the user a cleanup. An import that creates a prompt whose API key the destination workspace lacks leaves behind a copy that cannot run, and the user has to find that copy and delete it by hand. Running the checks first costs only the order of the steps. [#807 Export, import, compare, and verify prompts and assistants across LangSmith environments](https://github.com/codekiln/langstar/issues/807) asks import to check the destination's dependencies. + +A user therefore creates the destination workspace's secrets before importing a prompt, and imports the prompts before importing an assistant that refers to them. ### Users name an assistant's prompt references with `--prompt-ref` From 4eef77902a8be11e322c153b25045934fe21af8f Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Fri, 30 Oct 2026 09:53:08 -0400 Subject: [PATCH 86/90] =?UTF-8?q?=F0=9F=93=9A=20docs(openspec):=20send=20a?= =?UTF-8?q?n=20assistant's=20settings=20in=20one=20field=20on=20import?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A newer Agent Server refuses to create an assistant that sets both config.configurable and context, while every read returns both. Import now sends configurable alone when the two match, the one present when only one is, and stops with status 1 when they differ. Co-Authored-By: Claude Opus 5.5 --- .../design.md | 12 +++++++++ .../specs/assistant/spec.md | 27 +++++++++++++++++-- .../tasks.md | 4 ++- 3 files changed, 40 insertions(+), 3 deletions(-) diff --git a/openspec/changes/export-import-prompts-and-assistants/design.md b/openspec/changes/export-import-prompts-and-assistants/design.md index 7fabc3a62..ba3a90a3f 100644 --- a/openspec/changes/export-import-prompts-and-assistants/design.md +++ b/openspec/changes/export-import-prompts-and-assistants/design.md @@ -95,6 +95,18 @@ With `--update`, import makes an existing assistant match the file and asks the Without `--update`, import lists the differences and exits with status 3. `--dry-run` reports the action import would take and writes nothing. +### Import sends an assistant's settings in one field, never in both + +An assistant keeps its settings in two fields. `config.configurable` is the older one, and `context` arrived with LangGraph 0.6. When you read an assistant, the Agent Server returns the same settings in both fields, so an export file holds them twice. A newer Agent Server refuses to create an assistant that sets both, with status 400 and `Cannot specify both configurable and context. Prefer setting context alone.` An import that sent the file's fields as they stand would therefore fail on that server. + +So import sends one field: + +- When the file has both and they hold the same values, import sends `config` with `configurable` and leaves `context` out. The server fills `context` from `configurable` itself, so the read-back matches the file and compare finds no difference. A trial copy of assistants from one deployment to a newer one did exactly this. +- When the file has only one of them, import sends that one. +- When the file has both and they differ, import cannot tell which one the user meant. It writes nothing, names `/assistant/config/configurable` and `/assistant/context` without their values, and exits with status 1, the way it rejects a file of another kind. + +Export still writes both fields as the server returned them, so the file shows what the source held. `--update` sends the settings the same way. Task 1.2 records whether an update that sends both is refused too, and whether an update that sends `configurable` alone also replaces `context`. + ### Compare reports differences by JSON path Compare reads the prompt or assistant from the destination, puts it in the shape of the export file's `repo`, `commit` and `assistant` objects, and lists every value that differs from the file. Compare checks every key in those objects, including the chat model's settings and the `id` arrays inside a manifest that name the prompt and model classes, such as `ChatOpenAI`. It skips `commit.model_config` and `commit.model_provider`, because LangSmith fills both in itself when import pushes a commit, and a destination workspace can fill them in differently from the source. Export leaves the values the server sets, such as IDs, version numbers, timestamps, owner and like counts, out of those objects, so they never show up as differences. The source environment's workspace, deployment and IDs sit in `source`, so they never show up as differences against a copy. Each difference names its JSON Pointer path and both values. Before the report prints either side of a difference, it replaces with `` every string that export's credential check flags, by either rule, whether the difference is at that string or at an object that contains it. A credential someone pasted into the destination prompt or the destination assistant, or into the file, therefore stays out of the terminal and the CI log. The report prints a LangChain secret reference, such as `{"lc": 1, "type": "secret", "id": ["OPENAI_API_KEY"]}`, unchanged, because the reference holds only the secret's name, never its value. The report prints to a terminal or a CI log, where the user needs to see where each difference is, so it redacts the string and goes on reporting. For a prompt, the comparison covers the commit description, the commit tags and the commit each one points at, any tag on the compared commit that the file lacks, and the repo's description, readme, tags and visibility as well as the manifest. Import runs the same comparison after it writes, so a field LangSmith or the Agent Server drops shows up as a difference. diff --git a/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md b/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md index 4673f5b9a..11a0d3a16 100644 --- a/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md +++ b/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md @@ -129,7 +129,7 @@ A prompt name under `requires.prompts` with the owner `-`, such as `-/support-sy ### Requirement: Import an assistant into a deployment -With no created assistant ID recorded for the deployment and no `--assistant-id`, `langstar assistant import --deployment ` SHALL choose a new assistant ID, write it as pending in the destination record, then create the assistant with that `assistant_id`, `if_exists: do_nothing` and the file's name, description, graph ID, configuration, context and metadata, mark the entry created, and print the ID. A pending entry SHALL make the next import repeat the create. +With no created assistant ID recorded for the deployment and no `--assistant-id`, `langstar assistant import --deployment ` SHALL choose a new assistant ID, write it as pending in the destination record, then create the assistant with that `assistant_id`, `if_exists: do_nothing` and the file's name, description, graph ID, settings and metadata, mark the entry created, and print the ID. A pending entry SHALL make the next import repeat the create. #### Scenario: First import @@ -146,6 +146,29 @@ With no created assistant ID recorded for the deployment and no `--assistant-id` - **THEN** langstar sends the create with that ID and `if_exists` set to `do_nothing`, so the deployment still holds one assistant from the file - **AND** the destination record marks that ID created +### Requirement: Send an assistant's settings in one field + +When the file's `assistant` object holds both `config.configurable` and `context` with the same values, `langstar assistant import` SHALL send `config` with `configurable` and SHALL NOT send `context`. When it holds only one of them, import SHALL send that one. When it holds both and they differ, import SHALL write nothing, leave the destination record unchanged, name `/assistant/config/configurable` and `/assistant/context` without their values, and exit with status 1. + +#### Scenario: Both fields hold the same settings + +- **WHEN** the file's `config.configurable` and `context` both hold `{"system_prompt": "support-system:prod"}` +- **AND** a user runs `langstar assistant import support-bot.json --deployment my-agent-prod` +- **THEN** the create request holds `config.configurable` and no `context` +- **AND** the read-back's `context` holds `{"system_prompt": "support-system:prod"}`, so import reports no difference + +#### Scenario: Only context holds the settings + +- **WHEN** the file's `context` holds `{"system_prompt": "support-system:prod"}` and its `config` has no `configurable` +- **AND** a user runs `langstar assistant import support-bot.json --deployment my-agent-prod` +- **THEN** the create request holds `context` and no `config.configurable` + +#### Scenario: The two fields differ + +- **WHEN** the file's `config.configurable.system_prompt` is `support-system:prod` and its `context.system_prompt` is `support-system:staging` +- **AND** a user runs `langstar assistant import support-bot.json --deployment my-agent-prod` +- **THEN** langstar creates no assistant, leaves `support-bot.destinations.json` as it was, names `/assistant/config/configurable` and `/assistant/context` without their values and exits with status 1 + ### Requirement: Keep the assistant destination record at the path the user names `langstar assistant import`, `assistant compare` and `assistant verify` SHALL read and write the destination record at `.destinations.json` next to the export file, unless `--record ` names another file. @@ -192,7 +215,7 @@ When the destination record holds a created assistant ID for the deployment, `la ### Requirement: Update an existing assistant only on request -`langstar assistant import --update` SHALL update the recorded assistant, or the assistant named by `--assistant-id `, with the file's name, description, graph ID, configuration, context and metadata. `--assistant-id` SHALL take precedence over the destination record. +`langstar assistant import --update` SHALL update the recorded assistant, or the assistant named by `--assistant-id `, with the file's name, description, graph ID, settings and metadata, sending the settings as "Send an assistant's settings in one field" requires. `--assistant-id` SHALL take precedence over the destination record. #### Scenario: Update the recorded assistant diff --git a/openspec/changes/export-import-prompts-and-assistants/tasks.md b/openspec/changes/export-import-prompts-and-assistants/tasks.md index 27f1af996..d49e0e242 100644 --- a/openspec/changes/export-import-prompts-and-assistants/tasks.md +++ b/openspec/changes/export-import-prompts-and-assistants/tasks.md @@ -8,7 +8,7 @@ Each numbered group below is one pull request, merged in the order listed, excep - [ ] 1.2 Write `reference/research/807-openapi-validation.md`, recording what the live API returns. Verify that every claim in the report names the request that produced it. Without a second workspace, this task pushes to the existing test workspace under a fresh handle, and it runs a prompt with a model only when an OpenAI key is set. - For a structured prompt saved with a model, read with `include=model`: the manifest, its outer `id`, the `model_config` and `model_provider` beside it, and whether `model_provider` matches the class name export writes to `requires.model_class`. - Push that manifest to a second workspace: whether the read-back returns the same manifest, `model_config` and `model_provider`, and what pushing an identical manifest again returns. - - On the Agent Server: what `POST /assistants/search` returns for a graph ID the deployment does not serve, what it returns for `context`, whether an assistant update that sends a metadata key with a null value removes that key, whether an update that sends `config` or `context` replaces the assistant's object or merges into it, and whether an update can clear the description. + - On the Agent Server: what `POST /assistants/search` returns for a graph ID the deployment does not serve, what it returns for `context`, whether an assistant update that sends a metadata key with a null value removes that key, whether an update that sends `config` or `context` replaces the assistant's object or merges into it, and whether an update can clear the description. Also whether a create and an update that send both `config.configurable` and `context` are refused, and whether an update that sends `configurable` alone replaces `context` too. - What `POST /api/v1/prompts/invoke_prompt` returns for a prompt with a model. ## 2. Test environments @@ -154,9 +154,11 @@ Each integration test in this group carries `#[cfg_attr(not(feature = "integrati - Integration tests that a file whose graph ID the deployment lacks, and a file with a missing prompt tag, each create no assistant and exit with status 3. - A unit test in `cli/src/commands/assistant.rs` that serves, through a `mockito` server and `LangchainClient::with_base_urls`, an assistant read back that differs from the file, calls the import code twice, and asserts that the first call gives exit status 3 and leaves the new assistant ID in the destination record, and that the second call creates no assistant. - A CLI test that `assistant import` of a prompt export exits with status 1, names `kind` and the value `prompt`, and creates no assistant. + - Unit tests in `cli/src/commands/assistant.rs` that serve the Agent Server through a `mockito` server, one for each scenario of "Send an assistant's settings in one field" in `specs/assistant/spec.md`. They assert that a file with the same settings in both fields sends `config.configurable` and no `context`, that a file with only `context` sends `context` alone, and that a file whose two fields differ sends no request, leaves the destination record as it was, names both paths without their values and exits with status 1. - An integration test that creates an assistant with a chosen ID on the destination deployment, writes a destination record that holds that ID as pending, imports the file, and asserts that the deployment holds one assistant from the file, with that ID, and that the record marks the ID created. - [ ] 10.2 Add `--update`, `--assistant-id` and `--dry-run`. Verify with these tests: - An integration test that a repeated import creates no assistant. + - A unit test, through a `mockito` server, that `--update` of a file with the same settings in both fields sends `config.configurable` and no `context`. - An integration test that `--update` changes the recorded assistant's name, configuration, context and metadata on the existing test deployment, and a second integration test, carrying `#[ignore = "needs task 2.4"]`, that `--update` moves the recorded assistant from `test_graph` to `test_graph_alt` on the destination deployment from 2.4. - An integration test that `--update --assistant-id ` naming an assistant other than the recorded one updates that assistant, leaves the recorded one unchanged and writes the new ID to the destination record, and that `--assistant-id ` without `--update`, against an assistant that differs from the file, creates and changes no assistant and exits with status 3. - An integration test that `--update` of an assistant with metadata `{"kept": 1, "stale": 2}` from a file with `{"kept": 1}` leaves `{"kept": 1}`, or reports `/metadata/stale` and exits with status 3, whichever task 1.2 found the server does. From b1bf36aab7c1dd072801a945aff9ea84f5d4e42c Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Fri, 30 Oct 2026 10:09:11 -0400 Subject: [PATCH 87/90] =?UTF-8?q?=F0=9F=93=9A=20docs(openspec):=20rewrite?= =?UTF-8?q?=20the=20rest=20of=20design.md=20for=20the=20faults=20codekiln?= =?UTF-8?q?=20marked?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Same sweep as the verify and import-check sections, across the remaining decisions. The destination-record decision opened with the reason for a separate file instead of saying what import writes. The exit-status decision opened with a sentence that had to be read twice and ended in "neither success nor a broken run". The prompt-reference decision said the same thing in its first two sentences. The one-field heading read "in one field, never in both". The compare decision and the structured-output script were each one paragraph long enough to lose a reader; both are split by what they cover. Co-Authored-By: Claude Opus 5 --- .../design.md | 30 +++++++++++++------ 1 file changed, 21 insertions(+), 9 deletions(-) diff --git a/openspec/changes/export-import-prompts-and-assistants/design.md b/openspec/changes/export-import-prompts-and-assistants/design.md index ba3a90a3f..1685fbceb 100644 --- a/openspec/changes/export-import-prompts-and-assistants/design.md +++ b/openspec/changes/export-import-prompts-and-assistants/design.md @@ -13,7 +13,7 @@ Langstar reads one API key and one workspace at a time, from flags, environment **Goals:** - Recreate a prompt or an assistant from its export file with nothing lost. A prompt keeps its manifest and model, its commit description and tags, and its repo details. An assistant keeps its name, description, graph ID, configuration, context and metadata. -- Make a repeated import of the same file safe: it finds its earlier copy and changes nothing until the user asks. +- Make a repeated import of the same file safe: a second import of one file finds the copy the first one made and changes nothing until the user asks. - Give users one way to check a copy: the same report shape and exit statuses for both resource types. **Non-Goals:** @@ -77,7 +77,9 @@ Export reads the commit with `include=model`, the form the Go SDK uses (`referen ### Import records the IDs it created in a separate file next to the export -Keeping the record out of the export file means export never has to carry an earlier record forward when it overwrites the file, and the file a reviewer reads in Git changes only when the source changes. Import writes the IDs of what it created to `.destinations.json`, keyed by destination workspace ID for prompts and by deployment ID for assistants. The export file stays the same after any number of imports, and a fresh export never wipes out the record of earlier imports. Import replaces the record by writing the whole new record to a temporary file in the same directory and renaming it over the old one, so a langstar that stops in the middle of a write leaves the earlier record whole, pending assistant IDs included. Langstar supports one import of an export file at a time, because a person runs import by hand from a Git checkout. When two imports of the same file run at once, each reads the record before the other writes it, and the import that writes last drops the other's entries. Two first imports of one assistant into the same deployment can also create two assistants. [#828 make concurrent prompt and assistant imports into one deployment safe](https://github.com/codekiln/langstar/issues/828) covers making concurrent imports safe. +Import writes the IDs of what it created to `.destinations.json`, beside the export file, keyed by destination workspace ID for prompts and by deployment ID for assistants. A separate file keeps the two apart: export never has to carry an earlier record forward when it overwrites its file, the file a reviewer reads in Git changes only when the source changes, and a fresh export never wipes out the record of earlier imports. Import replaces the record by writing the whole new record to a temporary file in the same directory and renaming it over the old one, so a langstar that stops in the middle of a write leaves the earlier record whole, pending assistant IDs included. + +Langstar supports one import of an export file at a time, because a person runs import by hand from a Git checkout. When two imports of the same file run at once, each reads the record before the other writes it, and the import that writes last drops the other's entries. Two first imports of one assistant into the same deployment can also create two assistants. [#828 make concurrent prompt and assistant imports into one deployment safe](https://github.com/codekiln/langstar/issues/828) covers making concurrent imports safe. A repeated prompt import finds the copy by its handle and uses the record to tell whether someone changed it. Import writes the new commit hash to the record before it reads the commit back, so a read-back that finds a difference still leaves the hash in the record. The next import compares the destination's latest commit with that hash, and reports a change only when someone else pushed a commit after langstar's. Langstar can stop after it pushes a commit and before it writes that commit's hash to the record. The next import finds the destination's latest commit matching the file, writes its hash to the record and reports the prompt unchanged. @@ -95,13 +97,13 @@ With `--update`, import makes an existing assistant match the file and asks the Without `--update`, import lists the differences and exits with status 3. `--dry-run` reports the action import would take and writes nothing. -### Import sends an assistant's settings in one field, never in both +### Import sends an assistant's settings in one field -An assistant keeps its settings in two fields. `config.configurable` is the older one, and `context` arrived with LangGraph 0.6. When you read an assistant, the Agent Server returns the same settings in both fields, so an export file holds them twice. A newer Agent Server refuses to create an assistant that sets both, with status 400 and `Cannot specify both configurable and context. Prefer setting context alone.` An import that sent the file's fields as they stand would therefore fail on that server. +An assistant keeps its settings in `config.configurable`, the older field, and in `context`, which arrived with LangGraph 0.6. When langstar reads an assistant, the Agent Server returns the same settings in both, so an export file holds them twice. A newer Agent Server refuses to create an assistant that sets both, with status 400 and `Cannot specify both configurable and context. Prefer setting context alone.` An import that sent the file's fields as they stand would fail on that server. So import sends one field: -- When the file has both and they hold the same values, import sends `config` with `configurable` and leaves `context` out. The server fills `context` from `configurable` itself, so the read-back matches the file and compare finds no difference. A trial copy of assistants from one deployment to a newer one did exactly this. +- When the file has both and they hold the same values, import sends `config` with `configurable` and leaves `context` out. The server fills `context` from `configurable` itself, so the read-back matches the file and compare finds no difference. A trial run that copied assistants from one deployment to a newer one sent `config` this way, and every copy matched its file. - When the file has only one of them, import sends that one. - When the file has both and they differ, import cannot tell which one the user meant. It writes nothing, names `/assistant/config/configurable` and `/assistant/context` without their values, and exits with status 1, the way it rejects a file of another kind. @@ -109,7 +111,11 @@ Export still writes both fields as the server returned them, so the file shows w ### Compare reports differences by JSON path -Compare reads the prompt or assistant from the destination, puts it in the shape of the export file's `repo`, `commit` and `assistant` objects, and lists every value that differs from the file. Compare checks every key in those objects, including the chat model's settings and the `id` arrays inside a manifest that name the prompt and model classes, such as `ChatOpenAI`. It skips `commit.model_config` and `commit.model_provider`, because LangSmith fills both in itself when import pushes a commit, and a destination workspace can fill them in differently from the source. Export leaves the values the server sets, such as IDs, version numbers, timestamps, owner and like counts, out of those objects, so they never show up as differences. The source environment's workspace, deployment and IDs sit in `source`, so they never show up as differences against a copy. Each difference names its JSON Pointer path and both values. Before the report prints either side of a difference, it replaces with `` every string that export's credential check flags, by either rule, whether the difference is at that string or at an object that contains it. A credential someone pasted into the destination prompt or the destination assistant, or into the file, therefore stays out of the terminal and the CI log. The report prints a LangChain secret reference, such as `{"lc": 1, "type": "secret", "id": ["OPENAI_API_KEY"]}`, unchanged, because the reference holds only the secret's name, never its value. The report prints to a terminal or a CI log, where the user needs to see where each difference is, so it redacts the string and goes on reporting. For a prompt, the comparison covers the commit description, the commit tags and the commit each one points at, any tag on the compared commit that the file lacks, and the repo's description, readme, tags and visibility as well as the manifest. Import runs the same comparison after it writes, so a field LangSmith or the Agent Server drops shows up as a difference. +Compare reads the prompt or assistant from the destination, puts it in the shape of the export file's `repo`, `commit` and `assistant` objects, and lists every value that differs from the file. Each difference names its JSON Pointer path and both values. Compare checks every key in those objects, including the chat model's settings and the `id` arrays inside a manifest that name the prompt and model classes, such as `ChatOpenAI`. For a prompt, it also covers the commit description, the commit tags and the commit each one points at, any tag on the compared commit that the file lacks, and the repo's description, readme, tags and visibility. Import runs the same comparison after it writes, so a field LangSmith or the Agent Server drops shows up as a difference. + +Compare skips `commit.model_config` and `commit.model_provider`, because LangSmith fills both in itself when import pushes a commit, and a destination workspace can fill them in differently from the source. Export leaves the values the server sets out of those objects altogether — IDs, version numbers, timestamps, owner and like counts — and keeps the source environment's workspace, deployment and IDs in `source`, so none of them reads as a difference against a copy. + +The report goes to a terminal or a CI log, so it hides credentials before it prints. It prints `` in place of every string export's credential check flags, by either rule, whether the difference sits at that string or at an object holding it. A credential someone pasted into the destination prompt, the destination assistant or the file therefore stays off the screen while the user still sees where each difference is. A LangChain secret reference, such as `{"lc": 1, "type": "secret", "id": ["OPENAI_API_KEY"]}`, prints unchanged, because it holds the secret's name and never its value. ### `prompt verify` and `assistant verify` answer whether the copy can run @@ -137,15 +143,21 @@ A user therefore creates the destination workspace's secrets before importing a ### Users name an assistant's prompt references with `--prompt-ref` -A graph reads its prompt names from whatever configuration key its author chose, and guessing from strings that look like prompt handles would miss some and flag others, so the user says where they are. An assistant refers to a prompt through whatever configuration key its graph reads, so langstar cannot find the references by itself. `assistant export --prompt-ref ` names each one. The assistant's configuration should name each prompt by handle and tag, such as `support-system:prod`, because the same prompt gets a different commit hash in each workspace. Import copies the commit tags to the destination workspace, so when the graph on the destination deployment asks LangSmith for `support-system:prod`, it gets the same prompt. Compare, verify and import look up each pointer from `requires.prompts` in the file's `assistant` object, and exit with status 1 when the value there differs from the prompt name recorded beside the pointer. This catches a configuration someone edited in Git without updating `requires.prompts`. +An assistant refers to a prompt through whatever configuration key its graph reads, so langstar cannot find those references by itself, and guessing from strings that look like prompt handles would miss some and flag others. The user names each one with `assistant export --prompt-ref `. The assistant's configuration should name each prompt by handle and tag, such as `support-system:prod`, because the same prompt gets a different commit hash in each workspace. Import copies the commit tags to the destination workspace, so when the graph on the destination deployment asks LangSmith for `support-system:prod`, it gets the same prompt. Compare, verify and import look up each pointer from `requires.prompts` in the file's `assistant` object, and exit with status 1 when the value there differs from the prompt name recorded beside the pointer. This catches a configuration someone edited in Git without updating `requires.prompts`. ### compare, verify and import exit with 3 when they find a difference -Langstar exits today only with 0, with 1 from `main` in `cli/src/main.rs` when a command returns an error, and with clap's 2, so 3 is the first status that means neither success nor a broken run. Status 0 means the resource matches and every check passed. Status 3 means langstar found a difference or a failed check. Status 1 means any other error. Status 2 stays clap's: `Cli::parse()` in `run` in `cli/src/main.rs` exits with 2 when a user mistypes a flag or leaves out a required one, and this change leaves that alone, so a script never reads a typo as a difference. `prompt import` declares `--workspace-id` optional, as the other `langstar prompt` subcommands do, and checks for the flag itself. When a user leaves the flag out, langstar exits with status 1 and an error that names `--workspace-id`, where clap would have exited with 2. `diff` uses 1 for differences and 2 for errors; langstar keeps 1 for errors so that a script testing for failure sees the same status from every command. +Status 3 is new. Langstar exits today with 0 when a command succeeds, with 1 from `main` in `cli/src/main.rs` when a command returns an error, and with clap's 2, so 3 is the first status free to carry a meaning of its own. Status 0 means the resource matches and every check passed. Status 3 means langstar found a difference or a failed check. Status 1 means any other error. + +Status 2 stays clap's: `Cli::parse()` in `run` in `cli/src/main.rs` exits with 2 when a user mistypes a flag or leaves out a required one, and this change leaves that alone, so a script never reads a typo as a difference. `prompt import` declares `--workspace-id` optional, as the other `langstar prompt` subcommands do, and checks for the flag itself, so a user who leaves it out gets status 1 and an error naming `--workspace-id` where clap would have exited with 2. `diff` uses 1 for differences and 2 for errors; langstar keeps 1 for errors, so a script testing for failure sees the same status from every command. ### An optional Python script checks that an imported structured prompt still returns structured data -The LangSmith API has no documented way to run a prompt with its model, so a short Python script runs each imported structured prompt with LangChain and checks that the result matches the prompt's schema. The one prompt endpoint in the OpenAPI spec, `POST /api/v1/prompts/invoke_prompt`, takes message templates and inputs with no model or schema, and no LangSmith SDK calls it; in [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8), a developer calls it on the live API in case it does more than the spec says. A mise task runs the script with `uv run --locked --script` against a committed script lockfile, which installs the `langsmith` and `langchain-openai` packages the script lists, on a developer's machine and in the Integration Tests CI job. The script reads the workspace import wrote to: the import test passes it the workspace ID it gave `prompt import --workspace-id` and the handle it imported under, and the script builds its LangSmith `Client` with `workspace_id` set to that ID rather than reading `LANGSMITH_WORKSPACE_ID`. That is the existing test workspace today, and the destination workspace once [export-import-prompts-and-assistants Task 2.1 Add the destination workspace and the model provider key](tasks.md#L16) adds one. The script pulls the imported prompt with `include_model=True` and passes `secrets={"OPENAI_API_KEY": }` to [`pull_prompt`](https://github.com/langchain-ai/langsmith-sdk/blob/47c553caf08c1989e2be3ff1886da3eb801f709f/python/langsmith/client.py#L10228), because the LangSmith Python SDK reads no secret from the environment unless asked, and raises when a manifest holds a secret reference and no `secrets` map arrives. The script needs an OpenAI API key, which bills on every run. The check is optional: the mise task skips it and says so when `OPENAI_API_KEY` is unset, so every other test runs without a paid key, and a developer with a key of their own can run it by hand. +The LangSmith API has no documented way to run a prompt with its model, so a short Python script runs each imported structured prompt with LangChain and checks that the result matches the prompt's schema. The one prompt endpoint in the OpenAPI spec, `POST /api/v1/prompts/invoke_prompt`, takes message templates and inputs with no model and no schema, and no LangSmith SDK calls it. In [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8), a developer calls it on the live API in case it does more than the spec says. + +A mise task runs the script with `uv run --locked --script` against a committed script lockfile, which installs the `langsmith` and `langchain-openai` packages the script lists, on a developer's machine and in the Integration Tests CI job. The script reads the workspace import wrote to: the import test passes it the workspace ID it gave `prompt import --workspace-id` and the handle it imported under, and the script builds its LangSmith `Client` with `workspace_id` set to that ID rather than reading `LANGSMITH_WORKSPACE_ID`. That is the existing test workspace today, and the destination workspace once [export-import-prompts-and-assistants Task 2.1 Add the destination workspace and the model provider key](tasks.md#L16) adds one. + +The script pulls the imported prompt with `include_model=True` and passes `secrets={"OPENAI_API_KEY": }` to [`pull_prompt`](https://github.com/langchain-ai/langsmith-sdk/blob/47c553caf08c1989e2be3ff1886da3eb801f709f/python/langsmith/client.py#L10228), because the LangSmith Python SDK reads no secret from the environment unless asked, and raises when a manifest holds a secret reference and no `secrets` map arrives. Running the prompt bills an OpenAI API key on every run, so the mise task skips the check and says so when `OPENAI_API_KEY` is unset. Every other test runs without a paid key, and a developer with a key of their own runs this one by hand. ### Export and import use the same SDK calls as the planned `prompt create`, `get` and `update` From 442c0f6ef1e9f65f848c560d3636567d8abc3afd Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Fri, 30 Oct 2026 10:24:04 -0400 Subject: [PATCH 88/90] =?UTF-8?q?=F0=9F=93=9A=20docs(openspec):=20compare?= =?UTF-8?q?=20an=20assistant's=20settings=20from=20whichever=20field=20hol?= =?UTF-8?q?ds=20them?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A file that holds settings in config.configurable alone imported cleanly and then failed its own read-back: the Agent Server fills context from configurable, so the comparison found a context the file lacked and import exited with status 3. The same happened the other way round for a file that holds only context. Compare and the read-back now take the file's settings from whichever field the file holds and the destination's from whichever the server returned, and a field one side lacks is no difference. --update no longer sends an empty context beside a config.configurable. That would be refused by a newer server, or would wipe out the settings the server had just filled in. Import clears context only for a file that holds settings in neither field. New requirements and scenarios in specs/assistant for both rules, and tests in tasks 10.1 and 10.2 for a config-only file's round trip and for an --update that leaves the field it did not send alone. Co-Authored-By: Claude Opus 5 --- .../design.md | 8 +++- .../specs/assistant/spec.md | 46 ++++++++++++++++++- .../tasks.md | 5 +- 3 files changed, 54 insertions(+), 5 deletions(-) diff --git a/openspec/changes/export-import-prompts-and-assistants/design.md b/openspec/changes/export-import-prompts-and-assistants/design.md index 1685fbceb..2b3d15afb 100644 --- a/openspec/changes/export-import-prompts-and-assistants/design.md +++ b/openspec/changes/export-import-prompts-and-assistants/design.md @@ -93,7 +93,7 @@ Import treats the destination commit's tags as one more field to match. A tag in An import that stops after it creates the repo and before it pushes the commit leaves a prompt with no commit. The next import finds the prompt by its handle and, through the commit list, finds no commit. It then pushes the file's manifest as the first commit and creates the file's tags, as it does for a new prompt, without `--update`. The repo details it created hold the file's values already, and any that someone changed in between still need `--update`: without it, import pushes the commit and tags, leaves those repo details alone, lists them as differences and exits with status 3. -With `--update`, import makes an existing assistant match the file and asks the Agent Server to remove what the file lacks. The Agent Server keeps the name, graph ID and description an update leaves out, and merges the metadata an update sends into the metadata the assistant already has, as `AssistantPatch` in `reference/openapi/langchain/agent-server/openapi.json` documents. So import sends the file's name, graph ID, configuration and context in full, an empty object for a configuration or context the file lacks, and a null value for each metadata key the destination has and the file lacks. `AssistantPatch` documents no way to clear a description, so in [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8) a developer sends a null and an empty description to the live server, along with a null metadata value and a replaced configuration and context, and records what the server keeps. Import then reads the assistant back, and any key or field the file lacks that is still there shows up as a difference, with exit status 3. The Agent Server saves each update as a new version of the assistant. +With `--update`, import makes an existing assistant match the file and asks the Agent Server to remove what the file lacks. The Agent Server keeps the name, graph ID and description an update leaves out, and merges the metadata an update sends into the metadata the assistant already has, as `AssistantPatch` in `reference/openapi/langchain/agent-server/openapi.json` documents. So import sends the file's name, graph ID and settings in full, and a null value for each metadata key the destination has and the file lacks. [Import sends an assistant's settings in one field](#import-sends-an-assistants-settings-in-one-field) says which of the two settings fields import sends and which it leaves alone. `AssistantPatch` documents no way to clear a description, so in [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8) a developer sends a null and an empty description to the live server, along with a null metadata value and a replaced configuration and context, and records what the server keeps. Import then reads the assistant back, and any key or field the file lacks that is still there shows up as a difference, with exit status 3. The Agent Server saves each update as a new version of the assistant. Without `--update`, import lists the differences and exits with status 3. `--dry-run` reports the action import would take and writes nothing. @@ -107,7 +107,11 @@ So import sends one field: - When the file has only one of them, import sends that one. - When the file has both and they differ, import cannot tell which one the user meant. It writes nothing, names `/assistant/config/configurable` and `/assistant/context` without their values, and exits with status 1, the way it rejects a file of another kind. -Export still writes both fields as the server returned them, so the file shows what the source held. `--update` sends the settings the same way. Task 1.2 records whether an update that sends both is refused too, and whether an update that sends `configurable` alone also replaces `context`. +Compare reads an assistant's settings once, because the two fields hold one value. It takes the file's settings from whichever field the file holds and the destination's from whichever the server returned, and reports a difference only when the values differ. A settings field one side carries and the other lacks is no difference, since the server fills the second field from the first. So a file that holds settings only in `config.configurable` imports into a server that answers with both, and compare reports nothing. + +`--update` sends the settings in one field the same way, and leaves the other field alone. An empty `context` sent beside a `config.configurable` would either be refused by a newer server or wipe out the settings the server had just filled in from `configurable`. Import clears `context` only for a file that holds settings in neither field. In [export-import-prompts-and-assistants Task 1.2 Record what the live API returns](tasks.md#L8), a developer records whether an update that sends both is refused as a create is, and whether an update that sends `configurable` alone replaces `context` too. + +Export still writes both fields as the server returned them, so the file shows what the source held. ### Compare reports differences by JSON path diff --git a/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md b/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md index 11a0d3a16..c4622d27e 100644 --- a/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md +++ b/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md @@ -163,12 +163,41 @@ When the file's `assistant` object holds both `config.configurable` and `context - **AND** a user runs `langstar assistant import support-bot.json --deployment my-agent-prod` - **THEN** the create request holds `context` and no `config.configurable` +#### Scenario: Only config.configurable holds the settings + +- **WHEN** the file's `config.configurable` holds `{"system_prompt": "support-system:prod"}` and the file has no `context` +- **AND** a user runs `langstar assistant import support-bot.json --deployment my-agent-prod` +- **THEN** the create request holds `config.configurable` and no `context` + #### Scenario: The two fields differ - **WHEN** the file's `config.configurable.system_prompt` is `support-system:prod` and its `context.system_prompt` is `support-system:staging` - **AND** a user runs `langstar assistant import support-bot.json --deployment my-agent-prod` - **THEN** langstar creates no assistant, leaves `support-bot.destinations.json` as it was, names `/assistant/config/configurable` and `/assistant/context` without their values and exits with status 1 +### Requirement: Read an assistant's settings from whichever field holds them + +When the file holds settings in only one of `config.configurable` and `context`, `langstar assistant compare`, and the read-back import makes, SHALL compare the file's settings with the destination's settings from whichever field the destination holds, and SHALL NOT report the field the file lacks as a difference. + +#### Scenario: Import of a context-only file reports no difference + +- **WHEN** the file's `context` holds `{"system_prompt": "support-system:prod"}` and its `config` has no `configurable` +- **AND** the read-back's `config.configurable` holds `{"system_prompt": "support-system:prod"}` +- **THEN** import reports no difference and exits with status 0 + +#### Scenario: Import of a config-only file reports no difference + +- **WHEN** the file's `config.configurable` holds `{"system_prompt": "support-system:prod"}` and the file has no `context` +- **AND** the read-back's `context` holds `{"system_prompt": "support-system:prod"}` +- **THEN** import reports no difference and exits with status 0 + +#### Scenario: Compare of a config-only file against a destination holding both fields + +- **WHEN** the file's `config.configurable` holds `{"system_prompt": "support-system:prod"}` and the file has no `context` +- **AND** the destination assistant's `config.configurable` and `context` both hold `{"system_prompt": "support-system:prod"}` +- **AND** a user runs `langstar assistant compare support-bot.json --deployment my-agent-prod` +- **THEN** langstar reports no difference and exits with status 0 + ### Requirement: Keep the assistant destination record at the path the user names `langstar assistant import`, `assistant compare` and `assistant verify` SHALL read and write the destination record at `.destinations.json` next to the export file, unless `--record ` names another file. @@ -238,9 +267,9 @@ Without `--update`, `langstar assistant import --assistant-id ` SHALL compar - **AND** the user runs `langstar assistant import support-bot.json --deployment my-agent-prod --assistant-id ` - **THEN** langstar creates no assistant, changes none, lists the differences and exits with status 3 -### Requirement: Import with --update asks the Agent Server to remove the metadata, description and context the file lacks +### Requirement: Import with --update asks the Agent Server to remove the metadata and description the file lacks -`langstar assistant import --update` SHALL ask the Agent Server to remove each metadata key the destination assistant has and the file lacks, and to clear a description or context the file lacks, then read the assistant back. When such a key or field remains after the update, it SHALL list it as a difference and exit with status 3. +`langstar assistant import --update` SHALL ask the Agent Server to remove each metadata key the destination assistant has and the file lacks, and to clear a description the file lacks, then read the assistant back. When such a key or field remains after the update, it SHALL list it as a difference and exit with status 3. #### Scenario: Remove a metadata key the file lacks @@ -256,6 +285,19 @@ Without `--update`, `langstar assistant import --assistant-id ` SHALL compar - **WHEN** the Agent Server keeps `stale` in the assistant's metadata after the update - **THEN** langstar lists `/metadata/stale` as a difference and exits with status 3 +### Requirement: Import with --update leaves the settings field it did not send alone + +`langstar assistant import --update` SHALL clear `context` only when the file holds settings in neither `config.configurable` nor `context`. + +#### Scenario: --update of a config-only file leaves context alone + +- **WHEN** the file's `config.configurable` holds `{"system_prompt": "support-system:prod"}` and the file has no `context` +- **AND** the recorded assistant's `context` holds `{"system_prompt": "support-system:staging"}` +- **AND** the user runs `langstar assistant import support-bot.json --deployment my-agent-prod --update` +- **THEN** the update request holds `config.configurable` and no `context` +- **AND** the assistant's settings read back from the deployment hold `support-system:prod` +- **AND** langstar exits with status 0 + #### Scenario: Deployment keeps a description the file lacks - **WHEN** the recorded assistant has a description and the file has none diff --git a/openspec/changes/export-import-prompts-and-assistants/tasks.md b/openspec/changes/export-import-prompts-and-assistants/tasks.md index d49e0e242..edc7eb1c2 100644 --- a/openspec/changes/export-import-prompts-and-assistants/tasks.md +++ b/openspec/changes/export-import-prompts-and-assistants/tasks.md @@ -155,6 +155,8 @@ Each integration test in this group carries `#[cfg_attr(not(feature = "integrati - A unit test in `cli/src/commands/assistant.rs` that serves, through a `mockito` server and `LangchainClient::with_base_urls`, an assistant read back that differs from the file, calls the import code twice, and asserts that the first call gives exit status 3 and leaves the new assistant ID in the destination record, and that the second call creates no assistant. - A CLI test that `assistant import` of a prompt export exits with status 1, names `kind` and the value `prompt`, and creates no assistant. - Unit tests in `cli/src/commands/assistant.rs` that serve the Agent Server through a `mockito` server, one for each scenario of "Send an assistant's settings in one field" in `specs/assistant/spec.md`. They assert that a file with the same settings in both fields sends `config.configurable` and no `context`, that a file with only `context` sends `context` alone, and that a file whose two fields differ sends no request, leaves the destination record as it was, names both paths without their values and exits with status 1. + - Unit tests, through a `mockito` server, of the read-back for a file that holds settings in one field only: a file with `config.configurable` alone against a read-back that holds the same values in both fields, and a file with `context` alone against the same read-back. Each asserts that import reports no difference and exits with status 0. + - An integration test that imports an assistant from a file holding settings in `config.configurable` alone onto the test deployment, then runs `langstar assistant compare` and asserts status 0, so a file an older deployment produced copies onto a deployment that answers with both fields. - An integration test that creates an assistant with a chosen ID on the destination deployment, writes a destination record that holds that ID as pending, imports the file, and asserts that the deployment holds one assistant from the file, with that ID, and that the record marks the ID created. - [ ] 10.2 Add `--update`, `--assistant-id` and `--dry-run`. Verify with these tests: - An integration test that a repeated import creates no assistant. @@ -162,7 +164,8 @@ Each integration test in this group carries `#[cfg_attr(not(feature = "integrati - An integration test that `--update` changes the recorded assistant's name, configuration, context and metadata on the existing test deployment, and a second integration test, carrying `#[ignore = "needs task 2.4"]`, that `--update` moves the recorded assistant from `test_graph` to `test_graph_alt` on the destination deployment from 2.4. - An integration test that `--update --assistant-id ` naming an assistant other than the recorded one updates that assistant, leaves the recorded one unchanged and writes the new ID to the destination record, and that `--assistant-id ` without `--update`, against an assistant that differs from the file, creates and changes no assistant and exits with status 3. - An integration test that `--update` of an assistant with metadata `{"kept": 1, "stale": 2}` from a file with `{"kept": 1}` leaves `{"kept": 1}`, or reports `/metadata/stale` and exits with status 3, whichever task 1.2 found the server does. - - An integration test that `--update` of an assistant with a description and context from a file with neither clears both, or reports `/description` and `/context` and exits with status 3, whichever task 1.2 found the server does. + - An integration test that `--update` of an assistant with a description and settings from a file that holds neither a description nor settings in either field clears both, or reports `/description` and `/context` and exits with status 3, whichever task 1.2 found the server does. + - A unit test, through a `mockito` server, that `--update` of a file holding settings in `config.configurable` alone, against an assistant whose `context` holds other settings, sends `config.configurable` and no `context`, so the update never clears the field it did not send. - An integration test that a deleted recorded assistant exits with status 1. - An integration test that `--dry-run` creates no assistant, changes none and leaves the destination record as it was. - [ ] 10.3 Add an end-to-end test that imports prompts into the destination workspace, then imports an assistant that refers to them onto the destination deployment and runs `assistant verify`, passing `--workspace-id` set to the destination workspace to both assistant commands; verify that it passes, and that verify fails after deleting one prompt tag in the destination workspace through the tag delete call from 3.2 From dfd29d16b835f9eb3cbf47121fa725c77bec5bbc Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Fri, 30 Oct 2026 10:47:24 -0400 Subject: [PATCH 89/90] =?UTF-8?q?=F0=9F=93=9A=20docs(openspec):=20flag=20a?= =?UTF-8?q?=20short=20Basic=20credential,=20and=20read=20the=20settings=20?= =?UTF-8?q?alias=20in=20the=20shared=20comparison?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Copilot found two holes. The credential check asked for at least 16 base64 characters after `Basic `, and `Basic dXNlcjpwYXNz` carries twelve and decodes to `user:pass`. Under a key the name rule ignores, export would have written it to Git. The check now decodes the payload and flags it when the result is printable text with a colon in it, however short. `Basic instructions follow.` still passes, because its payload decodes to nothing readable. The settings alias lived in the assistant spec, while task 4.2's shared comparison still compared every key under `assistant`, so a one-field file would have been reported as a difference after all. Task 4.2 now reads `config.configurable` and `context` as one value and compares every other key under `config` as it stands, with tests for both single-field files and for a changed `config.recursion_limit`. Also a repeated-import test for each single-field file, and the matching scenario in specs/assistant. Co-Authored-By: Claude Opus 5 --- .../changes/export-import-prompts-and-assistants/design.md | 4 ++-- .../specs/assistant/spec.md | 7 +++++++ .../changes/export-import-prompts-and-assistants/tasks.md | 7 ++++--- 3 files changed, 13 insertions(+), 5 deletions(-) diff --git a/openspec/changes/export-import-prompts-and-assistants/design.md b/openspec/changes/export-import-prompts-and-assistants/design.md index 2b3d15afb..3cbddbd2e 100644 --- a/openspec/changes/export-import-prompts-and-assistants/design.md +++ b/openspec/changes/export-import-prompts-and-assistants/design.md @@ -63,7 +63,7 @@ Before it writes anything, export runs a credential check over every string it w The check flags a string when the name of its key or the shape of its value marks it as a credential. - **By key name:** the check flags a string whose key, or the key of the array it sits in, holds one of the words `secret`, `token`, `password`, `passwd`, `pwd`, `credential`, `credentials`, `apikey`, `authorization` or `cookie`, or holds `key` or `keys` right after `api`, `access`, `private`, `secret`, `signing` or `account`. It splits a key name into words at `_`, `-`, `.`, spaces and each change from a lower-case to an upper-case letter, and ignores case. It flags `openai_api_key`, `OPENAI_API_KEY`, `x-api-key`, `Authorization`, `access_token`, `client_secret`, `db_password`, `privateKey` and `api_keys`, and passes `max_tokens` and `cache_key`. -- **By value shape**, under any key: the check flags a string that holds a bearer token (`Bearer ` followed by at least 16 token characters) or a `Basic ` value followed by at least 16 base64 characters; a PEM private key (`-----BEGIN` and `PRIVATE KEY-----`); a URL with a password in it, `scheme://user:password@host` or `scheme://:password@host`, which covers database and queue connection strings; a key-value connection string with a non-empty `Password=`, `Pwd=`, `AccountKey=`, `SharedAccessKey=` or `SharedAccessSignature=`; a JSON Web Token, three base64url parts that start with `eyJ`; or a token with a provider's published prefix, which are `sk-` (OpenAI and Anthropic), `lsv2_` (LangSmith), `ghp_`, `gho_`, `ghs_` and `github_pat_` (GitHub), `xoxb-`, `xoxp-` and `xoxa-` (Slack), `AKIA` and `ASIA` followed by 16 upper-case letters or digits (AWS), `AIza` (Google) and `hf_` (Hugging Face), each followed by enough characters to rule out an ordinary word. +- **By value shape**, under any key: the check flags a string that holds a bearer token (`Bearer ` followed by at least 16 token characters) or a `Basic ` value whose base64 payload decodes to printable text with a colon in it, the `user:password` shape, however short that payload is: `Basic dXNlcjpwYXNz` carries twelve base64 characters and holds `user:pass`. A `Basic ` value whose payload decodes to anything else, as `Basic instructions follow.` does, passes; a PEM private key (`-----BEGIN` and `PRIVATE KEY-----`); a URL with a password in it, `scheme://user:password@host` or `scheme://:password@host`, which covers database and queue connection strings; a key-value connection string with a non-empty `Password=`, `Pwd=`, `AccountKey=`, `SharedAccessKey=` or `SharedAccessSignature=`; a JSON Web Token, three base64url parts that start with `eyJ`; or a token with a provider's published prefix, which are `sk-` (OpenAI and Anthropic), `lsv2_` (LangSmith), `ghp_`, `gho_`, `ghs_` and `github_pat_` (GitHub), `xoxb-`, `xoxp-` and `xoxa-` (Slack), `AKIA` and `ASIA` followed by 16 upper-case letters or digits (AWS), `AIza` (Google) and `hf_` (Hugging Face), each followed by enough characters to rule out an ordinary word. When the check flags any string, export writes no file and nothing to stdout, lists the JSON Pointer path of each flagged string with the rule that flagged it, such as `key name` or `URL with a password`, leaves the string and every part of it out of the message, and exits with status 1. Export refuses rather than writing `` in place of the credential. A redacted file would import a prompt or assistant that cannot authenticate, and a reviewer reading the file in a pull request would not see that the source holds a credential. The refusal asks the user to move the credential into the workspace's secrets, which a prompt's model reads through a secret reference, or into the deployment's environment variables, which a graph reads. @@ -115,7 +115,7 @@ Export still writes both fields as the server returned them, so the file shows w ### Compare reports differences by JSON path -Compare reads the prompt or assistant from the destination, puts it in the shape of the export file's `repo`, `commit` and `assistant` objects, and lists every value that differs from the file. Each difference names its JSON Pointer path and both values. Compare checks every key in those objects, including the chat model's settings and the `id` arrays inside a manifest that name the prompt and model classes, such as `ChatOpenAI`. For a prompt, it also covers the commit description, the commit tags and the commit each one points at, any tag on the compared commit that the file lacks, and the repo's description, readme, tags and visibility. Import runs the same comparison after it writes, so a field LangSmith or the Agent Server drops shows up as a difference. +Compare reads the prompt or assistant from the destination, puts it in the shape of the export file's `repo`, `commit` and `assistant` objects, and lists every value that differs from the file. Each difference names its JSON Pointer path and both values. Compare checks every key in those objects, including the chat model's settings and the `id` arrays inside a manifest that name the prompt and model classes, such as `ChatOpenAI`. It reads an assistant's `config.configurable` and `context` as one settings value, as [Import sends an assistant's settings in one field](#import-sends-an-assistants-settings-in-one-field) says, and compares every other key under `config` as it stands. For a prompt, it also covers the commit description, the commit tags and the commit each one points at, any tag on the compared commit that the file lacks, and the repo's description, readme, tags and visibility. Import runs the same comparison after it writes, so a field LangSmith or the Agent Server drops shows up as a difference. Compare skips `commit.model_config` and `commit.model_provider`, because LangSmith fills both in itself when import pushes a commit, and a destination workspace can fill them in differently from the source. Export leaves the values the server sets out of those objects altogether — IDs, version numbers, timestamps, owner and like counts — and keeps the source environment's workspace, deployment and IDs in `source`, so none of them reads as a difference against a copy. diff --git a/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md b/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md index c4622d27e..fca9a3528 100644 --- a/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md +++ b/openspec/changes/export-import-prompts-and-assistants/specs/assistant/spec.md @@ -191,6 +191,13 @@ When the file holds settings in only one of `config.configurable` and `context`, - **AND** the read-back's `context` holds `{"system_prompt": "support-system:prod"}` - **THEN** import reports no difference and exits with status 0 +#### Scenario: A second import of a config-only file creates nothing + +- **WHEN** the file's `config.configurable` holds `{"system_prompt": "support-system:prod"}` and the file has no `context` +- **AND** the destination record names the assistant an earlier import created, whose `config.configurable` and `context` both hold `{"system_prompt": "support-system:prod"}` +- **AND** a user runs `langstar assistant import support-bot.json --deployment my-agent-prod` +- **THEN** langstar creates no assistant, reports no difference and exits with status 0 + #### Scenario: Compare of a config-only file against a destination holding both fields - **WHEN** the file's `config.configurable` holds `{"system_prompt": "support-system:prod"}` and the file has no `context` diff --git a/openspec/changes/export-import-prompts-and-assistants/tasks.md b/openspec/changes/export-import-prompts-and-assistants/tasks.md index edc7eb1c2..cfb09a588 100644 --- a/openspec/changes/export-import-prompts-and-assistants/tasks.md +++ b/openspec/changes/export-import-prompts-and-assistants/tasks.md @@ -36,10 +36,11 @@ Each integration test in this group carries `#[cfg_attr(not(feature = "integrati - A test that updating one destination leaves the others unchanged. - A test that a record write that fails before the rename leaves the earlier record file unchanged, its pending entries included. - A test that reading a file whose `format_version` is 2, or whose `kind` names the other resource, returns an error naming the field and its value. -- [ ] 4.2 Add the JSON comparison, which compares every key under the `repo`, `commit` and `assistant` objects except `commit.model_config` and `commit.model_provider`, and reports differences by JSON Pointer. Verify with these tests: +- [ ] 4.2 Add the JSON comparison, which compares every key under the `repo`, `commit` and `assistant` objects except `commit.model_config` and `commit.model_provider`, reads `assistant.config.configurable` and `assistant.context` as one settings value, as [Import sends an assistant's settings in one field](design.md#import-sends-an-assistants-settings-in-one-field) says, and reports differences by JSON Pointer. Verify with these tests: - Unit tests on nested objects, arrays, missing keys, and keys holding `/` or `~`, which the report escapes as RFC 6901 requires, so `a/b` appears as `/a~1b` and `a~b` as `/a~0b`. - A test that a changed `id` array inside `commit.manifest`, such as `ChatOpenAI` replaced by `ChatAnthropic`, is reported at its path. - A test that a destination whose `commit.model_config` and `commit.model_provider` differ from the file's, while the manifest matches, is reported as a match. + - Tests of the settings alias: a file with `assistant.config.configurable` alone against a destination holding the same values in both fields, and a file with `assistant.context` alone against the same destination, each reported as a match; a file with `config.configurable` alone against a destination whose `context` holds other values, reported as one difference; and a test that every other key under `assistant.config` is compared as it stands, so a changed `config.recursion_limit` is reported at its path. - [ ] 4.3 Add the shared result report, which the compare, verify and import commands for both prompts and assistants print through, and the exit statuses 0, 3 and 1. The report prints JSON to stdout and progress to stderr. Before it prints either side of a difference, the report puts `` in place of every string the credential check from 4.4 flags, by either rule, and prints a secret reference unchanged under any key. It does this both when the difference is at the string and when it is at an object that contains it, for strings from the destination and from the file. Leave clap's status 2 for usage errors. Verify with these tests: - Unit tests of the report and of the exit status each result gives. - Redaction tests of a difference at `openai_api_key` where the destination holds a string; at `/commit/manifest/kwargs/last` where the file has no model and the destination's model holds a string under `openai_api_key`; at `/assistant/config/configurable/openai_api_key` and at `/assistant/config/configurable/OPENAI_API_KEY` where the destination holds a string; at a model's `default_headers/Authorization` holding `Bearer ` and a token; at a model's `model_kwargs/extra` holding a string that starts with `lsv2_`; at `/assistant/context/checkpoint_uri` holding a `postgresql://` URL with a password; and at `/assistant/config/configurable/db` where the destination adds `{"user": "app", "password": ""}`. Each asserts that the text and the `-f json` report print `` at the flagged string, that neither holds the string or any part of it, and, for the `db` object, that `user` prints unchanged. @@ -47,7 +48,7 @@ Each integration test in this group carries `#[cfg_attr(not(feature = "integrati - [ ] 4.4 Add the secret reference walk that lists secret names from a manifest, and the credential check that export, compare, verify and import share. The check walks every string under a JSON value, skips secret references and empty strings, and returns the JSON Pointer path of each flagged string with the rule that flagged it, the key-name rule or one of the value shapes in [Export refuses to write a file that holds a credential](design.md#export-refuses-to-write-a-file-that-holds-a-credential), and leaves the string out. Keep the word list and the value shapes in one module with a doc comment that names each. Verify with these tests: - Unit tests of the secret reference walk on a copy of the `prompt_with_model` manifest from the LangSmith Python SDK's integration tests, which `references/langsmith-sdk.md` links, and on a manifest with no secret reference. - Key-name tests that flag a string under `openai_api_key`, `OPENAI_API_KEY`, `x-api-key`, `Authorization`, `Cookie`, `access_token`, `refresh_token`, `client_secret`, `db_password`, `privateKey`, `signingKey` and `credentials`, and under an array whose key is `api_keys`, and that pass a string under `max_tokens`, `cache_key`, `author` or `keywords`, and a number or boolean under a flagged name. - - Value-shape tests under a neutral key such as `note` that flag `Bearer ` followed by a 20-character token, `Basic ` followed by 20 base64 characters, a PEM block for each of `PRIVATE KEY`, `RSA PRIVATE KEY` and `OPENSSH PRIVATE KEY`, `postgresql://app:@host/db`, `redis://:@host:6379`, `Server=db;User Id=app;Password=`, an Azure string with `AccountKey=`, a JSON Web Token, and a token with each prefix the design lists. Each asserts the rule named in the result. + - Value-shape tests under a neutral key such as `note` that flag `Bearer ` followed by a 20-character token, `Basic ` followed by 20 base64 characters, `Basic dXNlcjpwYXNz`, whose twelve base64 characters decode to `user:pass`, a PEM block for each of `PRIVATE KEY`, `RSA PRIVATE KEY` and `OPENSSH PRIVATE KEY`, `postgresql://app:@host/db`, `redis://:@host:6379`, `Server=db;User Id=app;Password=`, an Azure string with `AccountKey=`, a JSON Web Token, and a token with each prefix the design lists. Each asserts the rule named in the result. - Tests that pass ordinary text: `Basic instructions follow.`, `Bearer in mind`, `https://example.com/path`, `postgresql://host/db` with no password, `sk-` on its own, and `-----BEGIN PUBLIC KEY-----`. - Tests that skip a secret reference under `openai_api_key` and an empty string under `password`, and that escape a flagged key holding `/` or `~` in the path as RFC 6901 requires. - A test that the error message for each flagged string holds the path and the rule, and does not hold the string or any eight-character run of it. @@ -156,7 +157,7 @@ Each integration test in this group carries `#[cfg_attr(not(feature = "integrati - A CLI test that `assistant import` of a prompt export exits with status 1, names `kind` and the value `prompt`, and creates no assistant. - Unit tests in `cli/src/commands/assistant.rs` that serve the Agent Server through a `mockito` server, one for each scenario of "Send an assistant's settings in one field" in `specs/assistant/spec.md`. They assert that a file with the same settings in both fields sends `config.configurable` and no `context`, that a file with only `context` sends `context` alone, and that a file whose two fields differ sends no request, leaves the destination record as it was, names both paths without their values and exits with status 1. - Unit tests, through a `mockito` server, of the read-back for a file that holds settings in one field only: a file with `config.configurable` alone against a read-back that holds the same values in both fields, and a file with `context` alone against the same read-back. Each asserts that import reports no difference and exits with status 0. - - An integration test that imports an assistant from a file holding settings in `config.configurable` alone onto the test deployment, then runs `langstar assistant compare` and asserts status 0, so a file an older deployment produced copies onto a deployment that answers with both fields. + - An integration test that imports an assistant from a file holding settings in `config.configurable` alone onto the test deployment, runs `langstar assistant compare` and asserts status 0, then imports the same file again and asserts that it creates no second assistant and reports no difference. A file an older deployment produced therefore copies onto a deployment that answers with both fields, and stays matched on every import after the first. A second integration test does the same for a file holding settings in `context` alone. - An integration test that creates an assistant with a chosen ID on the destination deployment, writes a destination record that holds that ID as pending, imports the file, and asserts that the deployment holds one assistant from the file, with that ID, and that the record marks the ID created. - [ ] 10.2 Add `--update`, `--assistant-id` and `--dry-run`. Verify with these tests: - An integration test that a repeated import creates no assistant. From 531434bac4db6954e42e4c17b245df52a80c83ff Mon Sep 17 00:00:00 2001 From: codekiln <140930+codekiln@users.noreply.github.com> Date: Fri, 30 Oct 2026 11:10:08 -0400 Subject: [PATCH 90/90] =?UTF-8?q?=F0=9F=93=9A=20docs(openspec):=20keep=20a?= =?UTF-8?q?=20private=20prompt=20out=20of=20a=20public=20destination=20rep?= =?UTF-8?q?o?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Copilot found an exposure in the recovery path. When an earlier import left a repo with no commit and that repo is public, while the file says the prompt is private, import pushed the manifest and reported the visibility difference afterwards. By then the prompt's text was public, and no report takes that back. Import now pushes nothing in that case without --update, reports /repo/is_public and exits with status 3. With --update it makes the repo private before it pushes. The same holds for a repo someone else made. New requirement and scenarios in specs/prompt, and tests in tasks 7.1 and 7.2. Co-Authored-By: Claude Opus 5 --- .../design.md | 2 ++ .../specs/prompt/spec.md | 19 ++++++++++++++++++- .../tasks.md | 2 ++ 3 files changed, 22 insertions(+), 1 deletion(-) diff --git a/openspec/changes/export-import-prompts-and-assistants/design.md b/openspec/changes/export-import-prompts-and-assistants/design.md index 3cbddbd2e..6dc34d265 100644 --- a/openspec/changes/export-import-prompts-and-assistants/design.md +++ b/openspec/changes/export-import-prompts-and-assistants/design.md @@ -89,6 +89,8 @@ An assistant's ID differs on every deployment, so the record is the only way a r A prompt that exists in the destination gets a new commit only with `--update`, and its parent is the destination's latest commit, so LangSmith keeps the earlier history. Import sends the file's commit description with each commit it creates. With `--update`, import also sets the destination repo's description, readme, tags and visibility to the file's values through `PATCH /api/v1/repos/{owner}/{repo}` when they differ. When only those repo fields differ and the manifest matches, import updates the repo and adds no commit. +A public destination repo is the one repo detail import cannot leave for later. When the destination repo is public and the file's `repo.is_public` is false, a push would publish the prompt's text, and reporting the difference afterwards cannot take it back. So with `--update`, import makes the repo private before it pushes anything; without `--update`, it pushes nothing, reports `/repo/is_public` and exits with status 3. This holds for an import that finds a repo someone else made and for one finishing after an earlier run stopped between creating the repo and pushing its first commit. + Import treats the destination commit's tags as one more field to match. A tag in the file that the destination lacks, or that points at another commit there, is a difference, and so is a tag on the destination's matching commit that the file lacks. With `--update`, import creates or moves each of the file's tags to that commit and deletes each tag on it that the file lacks, through `DELETE /api/v1/repos/{owner}/{repo}/tags/{tag_name}`, so the read-back finds the same tags as the file. It adds no commit when the latest commit already matches. A tag on an older destination commit is not part of the comparison, so import leaves it alone. The tag endpoints take a commit's `id`, which differs from its hash. The commit read and the repo read return no `id`, so import looks the commit up in the commit list, `GET /api/v1/commits/{owner}/{repo}`, whose entries carry both the `id` and the hash. An import that stops after it creates the repo and before it pushes the commit leaves a prompt with no commit. The next import finds the prompt by its handle and, through the commit list, finds no commit. It then pushes the file's manifest as the first commit and creates the file's tags, as it does for a new prompt, without `--update`. The repo details it created hold the file's values already, and any that someone changed in between still need `--update`: without it, import pushes the commit and tags, leaves those repo details alone, lists them as differences and exits with status 3. diff --git a/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md index 0bd7ab355..ab919f310 100644 --- a/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md +++ b/openspec/changes/export-import-prompts-and-assistants/specs/prompt/spec.md @@ -193,7 +193,7 @@ When the destination workspace has no prompt with the file's handle, `langstar p ### Requirement: Finish an import that stopped before its first commit -When the destination workspace has a prompt with the file's handle and no commit, `langstar prompt import` SHALL push the file's manifest and commit description as the first commit and create the file's commit tags, as for a prompt that does not exist yet, without `--update`. Repo details that differ from the file SHALL still need `--update`: without it, import SHALL push the commit and tags, leave the repo details unchanged, list them as differences and exit with status 3. +When the destination workspace has a prompt with the file's handle and no commit, `langstar prompt import` SHALL push the file's manifest and commit description as the first commit and create the file's commit tags, without `--update`. Differing repo details SHALL still need `--update`: without it, import SHALL push the commit and tags, leave them unchanged, list them as differences and exit with status 3. "Keep a private prompt out of a public destination repo" covers a public one. #### Scenario: Earlier import created the repo and stopped @@ -202,6 +202,23 @@ When the destination workspace has a prompt with the file's handle and no commit - **THEN** langstar pushes the file's manifest as the prompt's first commit and creates the file's tags - **AND** exits with status 0 +### Requirement: Keep a private prompt out of a public destination repo + +When the destination repo is public and the file's `repo.is_public` is false, `langstar prompt import` without `--update` SHALL push no commit and no tag, report `/repo/is_public` and exit with status 3. With `--update` it SHALL make the repo private before it pushes the commit. + +#### Scenario: Public destination repo and no --update + +- **WHEN** the destination holds a repo named `my-prompt` that is public and has no commit +- **AND** the file's `repo.is_public` is false +- **AND** a user runs `langstar prompt import my-prompt.json --workspace-id ` +- **THEN** langstar pushes no commit and no tag, reports `/repo/is_public` and exits with status 3 + +#### Scenario: Public destination repo with --update + +- **WHEN** the same user runs the same import with `--update` +- **THEN** langstar sets the repo's visibility to private before it pushes the commit +- **AND** the prompt holds the file's manifest and the destination repo is private + ### Requirement: Change an existing prompt only with --update When the destination prompt exists and its latest commit differs from the file, `langstar prompt import` SHALL change nothing and exit with status 3, listing the differences, unless the user passes `--update`. With `--update` it SHALL push the file's manifest and commit description as a new commit whose parent is the destination's latest commit, and move the file's tags to it. diff --git a/openspec/changes/export-import-prompts-and-assistants/tasks.md b/openspec/changes/export-import-prompts-and-assistants/tasks.md index cfb09a588..3bd241687 100644 --- a/openspec/changes/export-import-prompts-and-assistants/tasks.md +++ b/openspec/changes/export-import-prompts-and-assistants/tasks.md @@ -96,6 +96,7 @@ Each integration test in this group carries `#[cfg_attr(not(feature = "integrati - A CLI test that runs import without `--workspace-id` while `LANGSMITH_WORKSPACE_ID` is set, and asserts that it exits with status 1, names the missing flag and creates no prompt. - A CLI test that `langstar -f json prompt import` of a new prompt prints one JSON object whose `action` is `create` and that holds the new commit hash, and that the same command with `--dry-run` prints `create` and a null commit hash. - An integration test that imports a file whose repo readme differs from the destination's, into a destination that holds the repo and no commit, without `--update`, and asserts that import pushes the commit and tags, leaves the readme unchanged, reports `/repo/readme` and exits with status 3. + - An integration test that creates a public repo with the file's handle and no commit in the destination workspace, then imports a file whose `repo.is_public` is false without `--update`, and asserts that the repo still holds no commit, that import reports `/repo/is_public` and exits with status 3, and that the repo is still the only thing there to find. - A CLI test that runs `prompt import` on an assistant export, and asserts that it exits with status 1, names `kind` and the value `assistant`, and creates no prompt. - CLI tests that import a file whose manifest refers to `ANTHROPIC_API_KEY` while `requires.secrets` holds only `OPENAI_API_KEY`, and a file whose `requires.model_class` names a class other than the one at the end of the manifest's chat model `id`, and assert that each creates no prompt, names the field and exits with status 1. - [ ] 7.2 Add `--update`, `--dry-run`, `--record` and the destination record. Verify with these tests: @@ -107,6 +108,7 @@ Each integration test in this group carries `#[cfg_attr(not(feature = "integrati - An integration test that `--update` adds one commit with the right parent and the file's commit description. - An integration test that pushes the file's manifest to the destination without the file's tags, as an import that stopped before creating its tags leaves it, runs import with `--update`, and asserts that it creates the missing tags on that commit and adds no commit. - An integration test that `--update` against a destination whose repo description, readme, tags or visibility differ while the manifest matches updates the repo and adds no commit. + - An integration test that `--update` against a public destination repo, from a file whose `repo.is_public` is false, makes the repo private before it pushes, asserted by the order of the two requests and by reading the repo back private with the file's commit. - An integration test that `--update` against a destination whose `prod` tag points at an earlier commit while the manifest matches moves the tag and adds no commit, and that `--dry-run` lists the move. - An integration test that `--update` against a destination whose latest commit matches the file and carries a `staging` tag the file lacks deletes `staging`, adds no commit and exits with status 0, and that `--dry-run` lists the delete. - An integration test that `--dry-run` writes nothing.