✨ feat(testing): pin the official LangChain tools and check prompt push with langsmith

docs/dev/testing/README.md

@@ -6,6 +6,7 @@ Load this TOC first, then 1-3 docs relevant to your task. **Do not load all docs
 |----------|--------------|
 | `HIGH_LEVEL_TESTING_GUIDELINES.md` | Always - PR requirements, andon cord |
 | `crud-lifecycle-pattern.md` | CRUD features - CLI & SDK verification |
+| `official-tool-checks.md` | Checking a langstar result with the official `langsmith` / `langgraph` tools |
 | `sdk-integration-tests.md` / `cli-integration-tests.md` | SDK or CLI integration tests |
 | `devcontainer-feature-tests.md` | Infrastructure/devcontainer testing |
 | `test-fixtures.md` | Test deployment & fixture management |

docs/dev/testing/official-tool-checks.md

@@ -0,0 +1,44 @@
+# Checking langstar with the official LangChain tools
+
+A langstar command can be wrong in a way its own SDK cannot see, because the CLI and SDK share the same request code. The official tools are a second implementation: create with langstar, read back with the official tool, and compare. This extends [crud-lifecycle-pattern.md](crud-lifecycle-pattern.md).
+
+## The tools
+
+`mise.toml` pins them; `mise install` installs them.
+
+| Tool | Provides |
+|------|----------|
+| [langsmith-cli](https://github.com/langchain-ai/langsmith-cli) | `langsmith` |
+| [langgraph-cli](https://pypi.org/project/langgraph-cli/) | `langgraph` |
+| [langgraph-sdk](https://pypi.org/project/langgraph-sdk/) | Python library; run it with `uv run --with "langgraph-sdk==$LANGGRAPH_SDK_VERSION"` |
+
+Bump a pin by hand and check that the release is older than five days: `minimum_release_age` does not apply to exact pins.
+
+## Which tool checks which command
+
+| langstar command group | Official tool that can read it back |
+|------------------------|-------------------------------------|
+| `prompt` | `langsmith prompt` |
+| `dataset` | `langsmith dataset`, `langsmith example` |
+| `project` | `langsmith project` |
+| `runs` | `langsmith run`, `langsmith trace` |
+| `eval` | `langsmith evaluator`, `langsmith experiment` |
+| `queue` | none |
+| `langsmith thread`, `langsmith hub` | no langstar command yet |
+| `deployment`, `assistant`, `graph` | `langgraph` (local server, build, `deploy list`/`delete`/`logs`/`revisions list`), `langgraph-sdk` (assistants, threads, runs on a deployment) |
+| `secrets`, `model-config`, `config` | none |
+
+The table says what a tool can reach, not what a check covers today. Run `langsmith <group> --help` to confirm a subcommand exists before writing a check.
+
+## Checks
+
+Each check is a mise task under `mise-tasks/oracle/`, named for the official tool acting as the reference result. It gets its credentials from `fnox exec` and cleans up what it creates.
+
+| Task | Checks |
+|------|--------|
+| `mise run oracle:prompt-roundtrip` | `langstar prompt push` against `langsmith prompt pull` |
+
+## Bugs the first check found
+
+- [#820](https://github.com/codekiln/langstar/issues/820): `langstar prompt push --owner -` to a new repo fails. The task creates the repo with `langsmith prompt create` first.
+- [#821](https://github.com/codekiln/langstar/issues/821): a push without `--schema` to an empty repo fails. The task pushes with `--schema`.

mise-tasks/oracle/prompt-roundtrip

@@ -0,0 +1,79 @@
+#!/usr/bin/env bash
+#MISE description="Push a private prompt with langstar, read it back with the official langsmith CLI, then delete it"
+set -euo pipefail
+
+# This task checks `langstar prompt push`. The official CLI reads the prompt
+# back, so a bug that langstar's push and pull share still shows up. The LangSmith
+# credentials come from fnox, so the check re-runs itself under `fnox exec`.
+# What each official tool can check: docs/dev/testing/official-tool-checks.md
+if [ -z "${ORACLE_UNDER_FNOX:-}" ] && { [ -z "${LANGSMITH_API_KEY:-}" ] || [ -z "${LANGSMITH_ORGANIZATION_ID:-}" ] || [ -z "${LANGSMITH_WORKSPACE_ID:-}" ]; }; then
+  ORACLE_UNDER_FNOX=1 exec fnox exec -- "$0" "$@"
+fi
+
+# langstar's prompt commands always call this endpoint (LANGSMITH_API_BASE in
+# sdk/src/client.rs), but the official CLI would otherwise take LANGSMITH_ENDPOINT or a
+# saved profile. --api-url on every official call keeps both tools on the same server.
+api_url="https://api.smith.langchain.com"
+official=(langsmith --api-url "$api_url")
+
+repo="oracle-roundtrip-$(date +%s)-$$-$RANDOM"
+template="Answer this question: {question}"
+# `-` is the owner handle of a private prompt in the workspace.
+handle="-/${repo}"
+
+# The official CLI creates the repo. langstar's own create-on-push sends "-/<repo>"
+# as the handle, which the API rejects, so letting langstar create it would fail here.
+# https://github.com/codekiln/langstar/issues/820
+"${official[@]}" prompt create --name "$repo" >/dev/null
+
+# Registered only after create succeeds, so a name collision can never delete
+# another run's prompt.
+schema=""
+# A failed delete turns a passing run into a failure and keeps an earlier failure's status.
+cleanup() {
+  local status=$?
+  [ -z "$schema" ] || rm -f "$schema"
+  if ! "${official[@]}" prompt delete --yes -- "$handle" >/dev/null 2>&1; then
+    echo "could not delete ${handle}; delete it by hand" >&2
+    [ "$status" -ne 0 ] || status=1
+  fi
+  exit "$status"
+}
+trap cleanup EXIT
+schema="$(mktemp)"
+
+printf '%s' '{"type":"object","title":"Out","properties":{"answer":{"type":"string"}},"required":["answer"]}' >"$schema"
+
+# --schema matches the path cli/tests/prompt_structured_test.rs exercises; a push
+# without it fails (https://github.com/codekiln/langstar/issues/821).
+cargo run --quiet --bin langstar -- prompt push \
+  --owner - --repo "$repo" --template "$template" --input-variables question \
+  --schema "$schema" >/dev/null
+
+pulled="$("${official[@]}" prompt pull -- "$handle")"
+
+# `langsmith prompt pull` prints {owner, repo, commit_hash, description, manifest}, and
+# the manifest is the StructuredPrompt in LangChain's JSON form that langstar sent
+# (sdk/src/prompts.rs). Check each pushed value at the field it belongs in, and compare
+# the schema with the file as a whole, so a changed type or a dropped `required` fails.
+# The message nesting matches what `langstar prompt pull` reads in cli/src/commands/prompt.rs.
+check() {
+  local what="$1"
+  shift
+  if ! printf '%s' "$pulled" | jq -e "$@" >/dev/null; then
+    echo "FAIL: the official CLI returned a different ${what} than langstar pushed" >&2
+    printf '%s\n' "$pulled" >&2
+    exit 1
+  fi
+}
+# The filters below are jq programs; `$want` and `$t` are jq variables, not shell ones.
+# shellcheck disable=SC2016
+{
+  check "prompt class" '.manifest.id[-1] == "StructuredPrompt"'
+  check "schema" --slurpfile want "$schema" '.manifest.kwargs.schema_ == $want[0]'
+  check "list of input variables" '.manifest.kwargs.input_variables == ["question"]'
+  check "message count" '.manifest.kwargs.messages | length == 1'
+  check "template" --arg t "$template" '.manifest.kwargs.messages[0].kwargs.prompt.kwargs.template == $t'
+  check "list of template variables" '.manifest.kwargs.messages[0].kwargs.prompt.kwargs.input_variables == ["question"]'
+}
+echo "OK: langsmith prompt pull returned the template, variables and schema pushed by langstar (${handle})"

mise.toml

@@ -37,6 +37,17 @@ age = "1.3.2"
 git-cliff = "2.14.2"
 # GitHub CLI, used by the worktree cleanup task and the issue workflow skills.
 gh = "2.102.0"
+# Official LangChain tools, used as an outside check on langstar's results
+# (docs/dev/testing/official-tool-checks.md). Exact pins skip minimum_release_age,
+# so each was chosen to be older than five days.
+# https://github.com/langchain-ai/langsmith-cli
+"github:langchain-ai/langsmith-cli" = "0.3.0"
+# langgraph-cli installs through uv; the LangGraph Python SDK is a library with no
+# executable, so tasks run it with `uv run --with langgraph-sdk==$LANGGRAPH_SDK_VERSION`.
+uv = "0.12.23"
+"pipx:langgraph-cli" = "0.4.33"
+# The oracle:* tasks parse the official CLI's JSON with jq.
+jq = "1.8.1"
 # rulesync is a Node CLI (v27 declares node >=22); without Node its executable
 # cannot run on a clean host.
 node = "22"
@@ -48,6 +59,8 @@ node = "22"
 enter = "lefthook install"
 
 [env]
+# The LangGraph Python SDK version that the oracle:* tasks will run; see [tools] above.
+LANGGRAPH_SDK_VERSION = "0.4.6"
 # Test fixture names are not secrets, so they live here rather than in 1Password.
 REPOSITORY_OWNER = "codekiln"
 REPOSITORY_NAME = "langstar"