You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
✨ feat(testing): pin the official LangChain tools and check prompt push with langsmith - #819
from AI(claude-code, claude-sonnet-5-5, langstar i747-oracles)
Refs #747 (install official LangChain tools through mise to check langstar's results)
Pins the official LangChain tools in mise.toml and adds one check that uses them against langstar. This is the small first PR; the rest of #747 (install official LangChain tools through mise to check langstar's results) follows.
Pins: github:langchain-ai/langsmith-cli 0.3.0 (release binary), pipx:langgraph-cli 0.4.33, uv 0.12.23, and LANGGRAPH_SDK_VERSION 0.4.6 in [env]. The LangGraph Python SDK is a library with no executable, so mise has nothing to install for it; check tasks run it with uv run --with. mise install works in a fresh worktree.
mise run oracle:prompt-roundtrip (mise-tasks/oracle/prompt-roundtrip): pushes a private prompt with langstar, reads it back with langsmith prompt pull, deletes it. Credentials through fnox exec. It passes.
docs/dev/testing/official-tool-checks.md: which tool can check which langstar command group.
mise run check passes.
Notes
The task is oracle:prompt-roundtrip because mise-tasks/check is already a file. The uv, langgraph-cli and SDK pins have no task using them yet; the assistant and deployment checks in the follow-ups will.
Copilot findings fixed
The task skipped fnox exec when only LANGSMITH_API_KEY was set: the guard now requires all three credentials.
The cleanup trap could delete another run's prompt after a name collision: it is now registered only after prompt create succeeds, and the name has a $RANDOM suffix.
jq was an undeclared dependency of the task: it is now pinned in mise.toml.
mktemp ran before the cleanup trap, so a failure there could leave the prompt behind: the trap is now installed first.
A failed cleanup delete was reported but the task still exited 0: it now exits non-zero and keeps an earlier failure's status.
Bugs the check found (not fixed here)
#820 (--owner - on a new repo) and #821 (push without --schema).
Decisions for codekiln
Task names start with oracle: (the official tool is the reference result). Recommend keeping them.
Pinned langsmith-cli 0.3.0, the newest release older than five days. Recommend bumping by hand.
…sh with langsmith
#747 746.2-oracles Install official LangChain tools through mise to check langstar's results
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…hat it created
#747 746.2-oracles Install official LangChain tools through mise to check langstar's results
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… doc and comments
#747 746.2-oracles Install official LangChain tools through mise to check langstar's results
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Unpinned jq dependency breaks oracle prompt roundtrip on fresh hosts
mise.toml:48
The new oracle invokes jq, but the supported host setup only requires mise and mise install, and jq is neither pinned here nor listed as a host prerequisite. A fresh host that follows the setup guide can therefore install every declared tool and still have oracle:prompt-roundtrip fail with jq: command not found. Pin jq alongside the other task dependencies (or replace the parsing step with the already-pinned Python runtime).
Update documentation to reference oracle checks instead of check tasks
mise.toml:60
This points readers to check:* tasks, but this PR deliberately places these checks under the oracle:* namespace. Keeping the old namespace here makes the pin's usage documentation misleading.
…rding
#747 746.2-oracles Install official LangChain tools through mise to check langstar's results
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This introduces an undeclared runtime dependency: mise install installs Python and the newly pinned LangChain tools, but not jq, so this task fails with jq: command not found on a clean non-devcontainer host. Please pin jq in mise.toml (or perform this validation with an already pinned runtime) so the documented task is self-contained.
A failed delete is converted to a successful echo, so a run can exit 0 while leaving the private prompt behind. Preserve the incoming status and change a previously successful run to failure when cleanup fails, while retaining an earlier test failure.
…tar uses
prompt-roundtrip passes --api-url on every official CLI call, so a
LANGSMITH_ENDPOINT or saved profile cannot send it to a different server than
langstar's push.
#747 Pin the official LangChain tools and check prompt push against them
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The reason will be displayed to describe this comment to others. Learn more.
🟢 Approval recommended
Static review found no blocking issues in the scoped tooling and verification task; live-service checks were not independently run.
No open issues 🎉
Review effort: Balanced
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
enhancementNew feature or requestpriority: mediumReview after high: bug fixes and features with no open decisionwipwork in progress
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
codekiln
added
ready-for-review
from AI(claude-code, claude-sonnet-5-5, langstar i747-oracles)
Refs #747 (install official LangChain tools through mise to check langstar's results)
Pins the official LangChain tools in
mise.tomland adds one check that uses them against langstar. This is the small first PR; the rest of #747 (install official LangChain tools through mise to check langstar's results) follows.github:langchain-ai/langsmith-cli0.3.0 (release binary),pipx:langgraph-cli0.4.33,uv0.12.23, andLANGGRAPH_SDK_VERSION0.4.6 in[env]. The LangGraph Python SDK is a library with no executable, so mise has nothing to install for it; check tasks run it withuv run --with.mise installworks in a fresh worktree.mise run oracle:prompt-roundtrip(mise-tasks/oracle/prompt-roundtrip): pushes a private prompt with langstar, reads it back withlangsmith prompt pull, deletes it. Credentials throughfnox exec. It passes.docs/dev/testing/official-tool-checks.md: which tool can check which langstar command group.mise run checkpasses.Notes
The task is
oracle:prompt-roundtripbecausemise-tasks/checkis already a file. Theuv,langgraph-cliand SDK pins have no task using them yet; theassistantanddeploymentchecks in the follow-ups will.Copilot findings fixed
fnox execwhen onlyLANGSMITH_API_KEYwas set: the guard now requires all three credentials.prompt createsucceeds, and the name has a$RANDOMsuffix.jqwas an undeclared dependency of the task: it is now pinned inmise.toml.mktempran before the cleanup trap, so a failure there could leave the prompt behind: the trap is now installed first.Bugs the check found (not fixed here)
#820 (
--owner -on a new repo) and #821 (push without--schema).Decisions for codekiln
oracle:(the official tool is the reference result). Recommend keeping them.langsmith-cli0.3.0, the newest release older than five days. Recommend bumping by hand.Follow-ups for #747
dataset(langsmith dataset,example),project,runs,eval(evaluator,experiment).langgraph-sdkcheck forassistantanddeploymentcommands.🤖 Generated with Claude Code