Repository navigation
Revive the cli ai commands on pydantic-ai 2.x with validated output and offline tests - #55
Merged
Merged
Conversation
- pydantic-ai is only imported when a command talks to a model, so the ai group is always registered (in cliffy.cli) and a missing `ai` extra is reported with an install hint instead of the group silently vanishing. - Model selection: --model, then CLIFFY_AI_MODEL, then a default for the first provider with an API key set. Unknown models and missing API keys exit with a readable error. - ai generate validates the reply (YAML, CLIManifest, generated code compiles) and sends failures back to the model before writing anything. It no longer overwrites an existing manifest without --force. - ai ask is grounded in the live command list, the cli init template and the manifest JSON schema instead of a hand-maintained copy. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Covers generate (valid, fenced, retry, exhausted retries, overwrite, unsafe names, failed request), ask (grounding, --cli reference), model selection, the missing-extra and missing-API-key paths, and that importing cliffy does not import pydantic-ai. Drops the W291 lint exemption cliffy/ai.py no longer needs and ignores a DeprecationWarning google-genai raises at import on Python 3.14+. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Reviewer's GuideRevives cli ai generate and ask for pydantic-ai 2.x by lazily resolving provider models, validating and retrying generated manifests before atomic-style write decisions, grounding answers in the installed cliffy package, and registering the commands with actionable errors and comprehensive offline tests. Sequence diagram for validated AI manifest generationsequenceDiagram
actor User
participant CLI as cli ai generate
participant Resolver as resolve_model
participant Agent as pydantic_ai Agent
participant Validator as validate_manifest_yaml
participant File as Manifest file
User->>CLI: generate description
CLI->>Resolver: resolve_model(model)
Resolver-->>CLI: provider model
CLI->>Agent: generate_manifest(description)
Agent->>Validator: output_validator(reply)
alt reply rejected
Validator-->>Agent: ModelRetry(reason)
Agent->>Validator: corrected reply
else manifest valid
Validator-->>Agent: validated YAML
end
Agent-->>CLI: manifest and usage
CLI->>File: write only after validation
CLI-->>User: success and token usage
Flow diagram for AI model selection and error handlingflowchart TD
Start["AI command"] --> Preview{"--preview?"}
Preview -->|yes| Prompt["Print generated prompt"]
Preview -->|no| Explicit{"--model or CLIFFY_AI_MODEL?"}
Explicit -->|yes| Model["Use selected provider:model"]
Explicit -->|no| Key{"Provider API key set?"}
Key -->|yes| Default["Use first provider default"]
Key -->|no| Fallback["Use google:gemini-flash-latest"]
Model --> Extra["Require pydantic-ai extra"]
Default --> Extra
Fallback --> Extra
Extra --> Error{"Model/provider usable?"}
Error -->|no| Hint["Exit with provider reason and install/key hint"]
Error -->|yes| Request["Send model request"]
File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
Contributor
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configuration
You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
There was a problem hiding this comment.
Hey - I've reviewed your changes and they look great!
Sourcery assessment
Needs a human reviewer. A generated manifest is written to disk and remains there if this change is reverted. The impact is bounded: the file can be inspected, edited, or deleted, and the CLI does not execute its generated code automatically.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
cli ai generateandcli ai askwork again on the current pydantic-ai (2.55 in the lockfile; theaiextra floor stayspydantic-ai>=2.0.0from the base branch). Model choice is provider-agnostic, generated manifests are validated before they are written, and both commands are tested offline. Stacked onchore/python-3.11-3.15.What changed
cliffy/ai.py(rewritten)Lazy import. pydantic-ai is only imported when a command actually talks to a model.
import cliffy.cliloads nopydantic_aimodule.Model selection. Order:
--model/-m, thenCLIFFY_AI_MODEL, then a default for the first provider whose API key is set:GOOGLE_API_KEYorGEMINI_API_KEYgoogle:gemini-flash-latestANTHROPIC_API_KEYanthropic:claude-sonnet-5-5OPENAI_API_KEYopenai:gpt-5.4-miniWith no key set it falls back to
google:gemini-flash-latest. All three ids are in the model list of the installed pydantic-ai 2.55.Clear errors. A missing extra exits with
pip install 'cliffy[ai]'. A missing API key, unknown model or missing provider SDK exits with the provider's reason plus a hint naming the key variables,--modelandCLIFFY_AI_MODEL. Any other import error is re-raised instead of being reported as a missing extra.ai generatevalidates before writing. The reply must be valid YAML, validate againstCLIManifest, use the requestedname, contain noincludes, and generate a CLI that compiles as Python (nothing is executed). A rejected reply is sent back to the model with the reason, up to--retriestimes (default 2). If no valid manifest comes back, the command fails and writes nothing.ai generateoptions.--forceis now required to overwrite an existing file. CLI names are restricted to letters, digits,-and_.ai askis grounded in the installed package. Its instructions are built from the live click command tree, the annotatedcli inittemplate andCLIManifest.model_json_schema().--cliaccepts a manifest file or a loaded CLI.Output. The answer goes to stdout and token usage to stderr.
--previewprints the prompt and needs neither the extra nor a key.--max-tokensis passed through as model settings.Registration (
cliffy/cli.py,cliffy/run.py)cli.pyimports theaigroup and registers it withcli.add_command(ai)(two lines; nothing else in the file changed).run.pyno longer importscliffy.aiinside a bareexcept Exception, which silently dropped the wholeaigroup on any import error. This file is outside the stated scope, but the old block is what hid the pydantic-ai API break.pyproject.tomlW291per-file ignore forcliffy/ai.py; the prompts no longer have trailing whitespace.filterwarningsignore for the'_UnionGenericAlias' is deprecatedwarning that google-genai raises at import on Python 3.14+.Tests and docs
tests/test_ai.pynow has 50 more tests (188 -> 238 in the suite), all offline, using pydantic-ai'sFunctionModelandTestModel. They cover generate (write, fence stripping, output dir, retry with the validation error, giving up,--force, unsafe names, failed request, empty reply,--max-tokens), ask (answer, grounding,--cliwith a file and a loaded CLI), missing extra, missing key, unknown model, missing provider SDK, and the no-import-at-startup guarantee.docs/features.md: the AI section documents model selection, validation and retries, grounding, and stdout/stderr behaviour.Breaking changes
cli aiis always registered. Without the extra, the commands exit with an install hint; previously the group was silently missing.google:gemini-2.0-flashto the per-provider table above.ai generateno longer overwrites an existing<name>.yamlwithout--force.ai generaterejects CLI names containing anything other than letters, digits,-and_.How verified
macOS arm64, uv 0.13.0,
uv sync --locked --all-extras, one environment per Python:Shell smoke tests on 3.15:
cli ai ask "hi"with no keys exits 1 with the missing-key message and hint.cli ai generate demo "say hello" -m testexits 1 and writes nothing.cli ai ask --preview "hi"prints the prompt.import cliffy.clileaves nopydantic_aimodule insys.modules.No request was sent to a real provider, so the default model ids are checked against pydantic-ai's model list only.
🤖 Generated with Claude Code
Summary by Sourcery
Restore reliable offline-tested AI commands with lazy model loading, validated manifest generation, and grounded CLI assistance.
New Features:
cli ai generateandcli ai askcommands with provider-agnostic model selection and optional dependency handling.ai askresponses in the installed cliffy command tree and manifest documentation, with support for manifest or loaded-CLI references.Bug Fixes:
aicommand group and provide actionable errors for missing extras, API keys, models, and provider SDKs.Enhancements:
Documentation:
Tests: