Skip to content

Fix audited defects: code generation escaping, env template injection, unsafe file writes, cli exit codes - #54

Merged
jaykv merged 14 commits into
mainfrom
fix/audit-bugs-and-security
Oct 12, 2026
Merged

jaykv merged 14 commits into
mainfrom
fix/audit-bugs-and-security

Conversation

@jaykv

@jaykv jaykv commented Oct 12, 2026 •

Copy link
Copy Markdown
Owner

Stack 3 of 6 (more to follow). Base: chore/python-3.11-3.15. Merge in order; each PR shows only its own changes.

Summary

Audit of cliffy/*.py and cliffy/commanders/*.py (not ai.py, blueprint.py or docs/), stacked on chore/python-3.11-3.15. Every fix below was reproduced first and has a regression test. The suite grows from 188 to 360 tests and passes with ruff and mypy --strict on Python 3.11 and 3.15.

The three most serious findings:

  • Template injection through the environment. Every value in os.environ was rendered by Jinja on each manifest load. An environment variable containing {{ ... }} was evaluated, and one containing {% (a zsh prompt, for example) made every manifest fail to load.
  • cli load could overwrite the interpreter. A manifest named python or pip wrote its entry script through the bin/python symlink, and cli rm then deleted it.
  • Manifest text was pasted into generated source unescaped. A quote, backslash or line break in a help text, version, var, default or alias gave a SyntaxError, a silently changed value, or code that ran when the CLI was imported.

Fixes

Severity is my assessment. Tests are in tests/.

# Severity Defect Fix Covered by
1 High Environment values were rendered as Jinja templates (injection, load failure on {%). {{ env.X }} also rendered empty outside vars. env is passed to templates as a mapping and never rendered. test_transformer.py::test_environment_values_are_not_rendered_as_templates, ::test_env_is_available_in_the_whole_manifest
2 High cli load overwrote, and cli rm deleted, any executable in the interpreter's bin directory with the CLI's name. Loading stops if the target exists and is not a cliffy entry script; unloading only removes cliffy entry scripts. test_loader.py::test_load_does_not_overwrite_foreign_executable, ::test_unload_keeps_foreign_executable
3 High Manifest strings (cli/command/group/param help, version, vars, config, cli_options, defaults, aliases, alias table) were interpolated between quotes in generated code. All strings go through py_str() / repr(). test_codegen.py::test_manifest_text_is_escaped (19 cases), ::test_manifest_text_is_not_executed (5), ::test_py_str_round_trips
4 High The CLI name was unchecked. name: ../../evil wrote the module, script and metadata outside their directories. Names are limited to letters, digits, _ and -, starting with a letter or _. test_transformer.py::test_cli_name_must_be_a_safe_file_name, ::test_cli_name_valid
5 Medium Commands named after a keyword, builtin or generated name (import, list, print, cli, typer, 2fa) broke the module or shadowed what other commands call. Groups and click aliases with a dash were SyntaxErrors. to_identifier() produces a safe function or variable name. An invalid param name is rejected with a clear error. test_codegen.py::test_names_become_valid_identifiers, ::test_keyword_param_name_is_rejected, ::test_to_identifier
6 Medium global_params were ignored for commands without params of their own. Applied to every command, in both generators. test_codegen.py::test_params[global_params_without_own_params], [click_global_params_without_own_params]
7 Medium name: str = typer.Option(...) was reduced to name: str, so examples/todo.yaml's prompt was lost. A default containing = was truncated. --dry-run|-d became --dry_run. A non-str default on a str param raised AttributeError. Parser fixes in parser.py and manifest.py. test_codegen.py::test_params (typer cases), ::test_manifest_text_is_escaped[param_default_int_for_str]
8 Medium Greedy (*) commands ignored their template and | aliases, were attached to the wrong nested group, and crashed on non-str defaults. Template and aliases are resolved for lazy commands; group lookup uses rsplit(".", 1). test_codegen.py::test_greedy_commands (5 cases)
9 Medium Click generator: --name|-n, name: --who (became ----who), dashed options/arguments and global params produced broken code; bool options needed a value. ClickParser rewritten on top of Parser. test_codegen.py::test_params (click cases)
10 Medium A script-less a.b command carrying help for nested group a.b generated an empty function (SyntaxError, Typer). Skipped, as root commands already were. test_codegen.py::test_names_become_valid_identifiers[nested_group_help]
11 Medium requires validation: PyYAML>=6 raised KeyError, requests>=2,<3 raised InvalidVersion, and ~=, extras, rich_click and markers were reported as missing. Parsed with packaging and checked against importlib.metadata, with normalized names. test_transformer.py::test_requires_met, ::test_requires_unmet; test_helper.py::test_check_requirement (18 cases)
12 Medium Includes: relative paths resolved against the working directory; merge order went through a set(); a cycle ended in RecursionError. Looked up next to the manifest first (cwd as fallback), merged in order, cycles reported. test_transformer.py::test_includes_*, ::test_circular_includes, ::test_missing_include
13 Medium cli test exited 0 when tests failed (also with -x) and errored on every test of a use_click manifest. Exits 1 unless all pass; uses click's runner for click CLIs. test_tester.py::test_manifest_tests_*
14 Medium A zipapp built from a use_click manifest did not bundle rich-click. Framework requirement is read from the generated code. test_builder.py::test_build_installs_the_framework_the_cli_imports
15 Medium With LANG=C, cli load failed with UnicodeEncodeError on any non-ASCII manifest text. Generated code, manifests and metadata are read and written as UTF-8. test_encoding.py (2 tests, run in a C-locale subprocess)
16 Medium cli docs <loaded-cli> always failed; cli build <loaded-cli> looked for my-cli.py instead of my_cli.py. Docs transform the source manifest; build uses Loader.get_cli_path. test_cli.py::test_docs_for_loaded_cli, ::test_build_loaded_cli_with_dashed_name
17 Low cli validate, build, update, remove and docs exited 0 after printing an error. Exit 1. test_cli.py::test_validate_reports_generation_errors_with_exit_code, ::test_build_failure_exit_code, ::test_docs_for_unknown_cli, plus 4 updated assertions
18 Low cli rm '*' deleted other CLIs' metadata; remove_metadata raised if the directory held another file; a metadata file handle was leaked. Literal name match, tolerant rmdir, context manager. test_homer.py::test_remove_metadata_*, ::test_get_metadata_bypath_closes_file
19 Low unload_cli never removed the module when the script was already gone. Separate suppress blocks. test_loader.py::test_unload_removes_module_when_script_is_missing
20 Low Temp scripts were left behind when a build or import failed; cli build -o a/b failed if a did not exist. try/finally; os.makedirs. test_builder.py::test_*_removes_temp_file_*, ::test_build_creates_nested_output_dir; test_tester.py::test_tester_removes_temp_file_when_import_fails
21 Low Reloader: watching x.yaml also reloaded on other-x.yaml; the manifest was never closed; the observer was only stopped on Ctrl-C. Full path comparison, context manager, finally. test_reloader.py::test_on_modified_only_reloads_the_watched_manifest, ::test_reload_closes_manifest, ::test_watch_stops_observer_on_unexpected_error
22 Low Empty, list or scalar manifests and an empty vars: ended in AttributeError/TypeError tracebacks. Error message and exit 1. test_transformer.py::test_manifest_must_be_a_mapping, ::test_empty_vars_section
23 Low cli docs -f html did not escape manifest text; examples without a description printed "None". html.escape; empty string. test_doc.py::test_generate_html_escapes_manifest_text, ::test_example_without_description
24 Low The fallback Table (no rich-click) could only be rendered once. Styles are no longer popped. test_rich.py::test_fallback_table_can_be_rendered_twice
25 Low cli init printed a traceback for a name the template rejects; --raw accepted names that cannot be loaded. Name check and clean error. test_cli.py::test_init_invalid_name, ::test_init_template_name_error_is_not_a_traceback
26 Low Generated base imports came out in set order, so output differed between runs. Sorted. test_codegen.py::test_generated_imports_are_sorted

Each new test was also run against the base commit's code to confirm it fails there. A handful pass on base on purpose: they guard behavior that must not change (for example test_load_replaces_its_own_script, test_params[click_quoted_default], test_cli_name_valid). test_generated_imports_are_sorted failed on base in my runs, but because the old order was hash-dependent it could pass there by chance.

Breaking changes

  1. CLI names are restricted to [A-Za-z_][A-Za-z0-9_-]*. Names with spaces, dots or slashes are rejected.
  2. Exit codes. cli test exits 1 when any test fails. cli validate, build, update, remove and docs exit 1 on error. Four existing assertions that pinned exit code 0 for these cases were changed to 1 (test_cli.py::test_update_nonexistent_cli, ::test_remove_command, ::test_build_command, test_examples.py::test_cli_build_fails).
  3. Generated function names. A command named after a keyword, builtin or generated name gets a trailing _ (list -> list_). Run blocks that call such a command's function by name must be updated. Command names on the command line are unchanged.
  4. global_params now apply to every command, including those without params of their own.
  5. Click generator: bool options are --flag/--no-flag switches (was --flag true). A bare str default (str = World) is a string.
  6. Typer generator: an option with a short alias keeps its dashes (--dry-run, was --dry_run). Simple params written as a full typer.Option(...) / typer.Argument(...) are now honored, so todo create prompts as its manifest says.
  7. Includes are resolved next to the including manifest first, and are merged in the order given.
  8. {{ env.X }} works outside vars (it rendered empty before).
  9. requires entries must be valid PEP 508 requirements. Markers are evaluated.
  10. cli load refuses a name that collides with an existing non-cliffy executable in the interpreter's bin directory.
  11. Helper API: parse_requirement, compare_versions, RequirementSpec and OPERATOR_MAP are removed in favor of check_requirement. get_installed_package_versions returns normalized names from importlib.metadata. Transformer.resolve_include_by_path is replaced by resolve_include. The tests for the removed helpers were ported case by case to test_check_requirement.

Looked at and left alone

  • Run blocks, imports, functions, types, raw param strings, simple-param defaults and the name: {expr: null} var form are code by design.
  • Jinja is not sandboxed and uses the default Undefined. The manifest author is trusted, and StrictUndefined would break the documented {{ env['X'] or 'default' }} pattern.
  • YAML is loaded with yaml.safe_load everywhere.
  • Includes can point at any readable YAML file. That is the author's choice, not traversal.
  • builder.py calls pip and shiv with argument lists (no shell) and cleans its TemporaryDirectory.
  • tester.py execs the manifest's assert scripts and shell lines by design.
  • Mutable defaults on pydantic models are copied per instance; build_cli(deps=None) is handled.
  • Reload threads are daemons; an error in one prints and the watcher keeps running.
  • The pybash $ transform was not audited beyond the existing example tests.

How verified

Local, macOS arm64, uv 0.13.0, one venv per Python synced with uv sync --locked --all-extras:

Python ruff format --check ruff check mypy cliffy --strict mypy tests pytest
3.11.11 pass pass pass pass 360 passed
3.15.0 pass pass pass pass 360 passed
  • Every commit was exported and checked on 3.11: tests and mypy --strict pass on all 13. Ruff passes on 12; the first commit has three unused imports that the second commit starts using.
  • Not run: Python 3.12-3.14, Linux, Windows, and GitHub CI.

🤖 Generated with Claude Code

Summary by Sourcery

Harden manifest processing, code generation, CLI lifecycle operations, and command outcomes against audited defects and unsafe input.

Bug Fixes:

  • Prevent environment values from being recursively rendered as templates and make environment variables available throughout manifests.
  • Prevent generated code from being corrupted or executing unintended manifest text by safely escaping generated strings and identifiers.
  • Prevent CLI loading and unloading from overwriting or deleting foreign executables.
  • Correct manifest includes, requirement validation, parameter parsing, greedy commands, documentation rendering, temporary-file cleanup, and reload behavior.
  • Return failure exit codes for failed tests and CLI operations.

Enhancements:

  • Strengthen manifest and CLI name validation, UTF-8 handling, metadata cleanup, generated import ordering, and fallback table rendering.
  • Improve support for Typer and Click parameters, aliases, global parameters, nested commands, and loaded CLI documentation and builds.

Build:

  • Ensure builds package the framework used by generated CLIs and create nested output directories.

Documentation:

  • Escape manifest content in generated HTML documentation and omit missing example descriptions.

Tests:

  • Add comprehensive regression coverage for code generation safety, manifest transformation, loading, building, testing, documentation, encoding, and CLI exit codes.

Chores:

  • Replace ad hoc requirement parsing with PEP 508-aware dependency checks and remove obsolete helper APIs.

jaykv and others added 14 commits October 12, 2026 06:08
Requirements were parsed by hand: one operator, matched by substring, and
the installed versions came from `pip freeze`. As a result
- `PyYAML>=6` raised KeyError (the name was lowercased for one lookup only),
- `requests>=2,<3` raised InvalidVersion,
- `click~=8.0`, `typer[all]>=0.9`, `rich_click` and requirements with an
  environment marker were reported as missing,
- packages that `pip freeze` does not print as `name==version` (editable
  installs) or environments without pip could not be validated at all.

Requirements are now parsed with packaging.requirements.Requirement and
checked against importlib.metadata, with PEP 503 name normalization. The
error message also gets its missing line break.

parse_requirement, compare_versions, RequirementSpec and OPERATOR_MAP are
replaced by check_requirement; their tests are ported to it case by case.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Help texts, versions, vars, defaults, aliases and command config were
pasted between quotes in the generated source. A quote, a backslash or a
line break in any of them produced a SyntaxError, silently changed the
value (`C:\temp\new` became a tab and a line break), or ran as code when
the CLI was imported. Every string now goes through py_str()/repr().

Names are turned into identifiers that are safe to define:
- commands named like a keyword, a builtin or a name the generated module
  uses itself (`import`, `list`, `print`, `cli`, `typer`, `2fa`) no longer
  break the module or shadow what other commands call,
- groups and click aliases with a dash (`my-group.sub`, `rm-all`) work,
- a param that is not a valid Python variable name is rejected with a
  clear message instead of a SyntaxError at load time.

Param handling:
- global_params are applied to commands without params of their own,
- a default may contain `=` (`--who: str = "a=b"`),
- `name: str = typer.Option(...)` keeps its typer definition instead of
  being reduced to `name: str`,
- `--dry-run|-d` keeps the option name `--dry-run` (was `--dry_run`),
- a non-str default of a str param no longer raises AttributeError.

Greedy `(*)` commands now get their template and `|` aliases, are attached
to the right nested group, and accept non-str defaults.

Click generator: `--name|-n`, `name: --who`, dashed options and arguments
and global params generate working decorators; bool options are
`--flag/--no-flag` switches as in the Typer generator. ClickParser now
extends Parser instead of copying it.

The base imports are written in sorted order, so the generated code no
longer differs from run to run. A script-less `a.b` command that only
carries the help of the nested group `a.b` no longer generates an empty
function.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The manifest `name` becomes the file name of the generated module, of the
entry script in the interpreter's bin directory and of the metadata file,
and it is written into the import line of that script. Nothing checked
it: `name: ../../evil` wrote files outside those directories, and a name
with a space, a dot or a line break produced a script that cannot run.

Names are now limited to letters, digits, `_` and `-`, starting with a
letter or `_`, which covers every name that could be loaded and run so far.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ifests

`env` was put into the vars that are interpolated on every load, so the
whole of os.environ was rendered by Jinja as if it were a template:
- an environment variable holding `{{ ... }}` was evaluated, which is
  template injection through the environment of whoever runs `cli load`,
- one holding `{%` (a zsh prompt, for example) made every manifest fail
  with TemplateSyntaxError,
- the result replaced `env` with a string, so `{{ env['X'] }}` only
  worked inside `vars` and rendered empty everywhere else.

`env` is now handed to the templates as the mapping it is, for `vars` and
for the rest of the manifest alike.

An empty manifest, a manifest that is a list or a scalar, and an empty
`vars:` section now end in an error message instead of an
AttributeError/TypeError traceback. The manifest is read as UTF-8.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… cycles

- Relative include paths were resolved against the working directory, so
  a manifest only loaded from the directory it was written for. They are
  now looked up next to the including manifest first; the working
  directory stays as a fallback for existing manifests.
- The includes went through a set() before merging. The first manifest to
  define a key wins, so with conflicting includes the result depended on
  the hash seed of the process. The documented order is now kept.
- Two manifests including each other recursed until RecursionError. The
  chain of manifests being resolved is tracked and a cycle is reported.
- A missing include is reported with the manifest that asked for it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Files were opened with the locale encoding. With LANG=C (containers, cron,
CI) `cli load` failed with UnicodeEncodeError as soon as a manifest held a
non-ASCII character, and a manifest could be decoded differently by the two
passes that read it. Python reads the generated source as UTF-8, so that is
what gets written.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- `cli rm '*'` (or any name with glob characters) matched the metadata of
  other CLIs and deleted it, leaving their scripts and modules behind with
  nothing left to remove them by. Names are now matched literally.
- remove_metadata raised OSError when the metadata directory held another
  file (a .DS_Store, for example) after the metadata file was already
  deleted. The directory is only removed once it is empty.
- get_metadata_bypath never closed the metadata file.
- Metadata and manifests are read as UTF-8.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`cli load` writes an entry script named after the CLI into the bin
directory of the interpreter. A manifest named `python`, `pip` or after
any other installed tool replaced that executable: bin/python is a symlink,
so the write went through it and overwrote the interpreter itself, and
`cli rm` deleted it afterwards. Loading now stops with an error when the
target exists and is not a cliffy entry script, and unloading only removes
cliffy entry scripts.

unload_cli also wrapped both removals in one suppress(): when the script
was already gone, the generated module was never removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- A zipapp built from a `use_click` manifest only bundled typer and failed
  with "No module named 'rich_click'" on any machine that does not have
  rich-click installed. The framework is now read from the generated code.
- The temporary script was only deleted on success: a failed build or a
  generated module that failed to import left it in the temp directory.
- `cli build -o a/b` failed when `a` did not exist.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- `cli test` exited 0 when tests failed, also with `-x`, so it could not
  gate anything in CI. It now exits 1 unless every test passed.
- `cli test` errored on every test of a `use_click` manifest: typer's
  runner cannot invoke a click group. The click runner is used for those.
- `cli validate` exited 0 for a manifest that failed code generation.
- `cli build`, `cli update` and `cli remove` exited 0 after printing an
  error for a CLI that is not loaded or a build that failed. They still
  process the remaining arguments, then exit 1.
- `cli build <loaded-cli>` looked for `my-cli.py`, but the module of a
  loaded CLI is stored as `my_cli.py`.
- `cli docs <loaded-cli>` validated the raw manifest text as a model and
  always failed; an unknown name printed nothing and exited 0.
- `cli init` printed a traceback for a name the template rejects, and
  `--raw` accepted names that can never be loaded.
- The tester left its temporary module behind when the import failed.
- `exit()` (a site builtin that is missing under `python -S`) is no longer
  used; `__main__` had an unreachable one.

Four existing assertions that pinned exit code 0 for those error cases are
updated to 1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The changed path was compared with endswith(), so watching `x.yaml` also
  reloaded on changes to `other-x.yaml` in the same directory. Paths are
  compared in full.
- reload() never closed the manifest it opened, once per reload.
- The observer thread was only stopped on KeyboardInterrupt.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Without rich-click, Table.__str__ popped the column styles while rendering,
so a second str() of the same table raised IndexError.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`cli docs -f html` pasted help texts, params and examples into the page
as-is, so `<src>` vanished and markup in a manifest ended up in the page.
An example given as a plain string has no description and was followed by
the text "None" in all three formats.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sourcery-ai

sourcery-ai Bot commented Oct 12, 2026 •

Copy link
Copy Markdown

Reviewer's Guide

This audit hardens Cliffy's manifest-to-code pipeline by escaping all data-derived source, isolating environment values, validating names and filesystem operations, and repairing parser behavior; it also makes dependency/include resolution, encoding, cleanup, documentation, testing, builds, reloads, and CLI exit codes reliable, with broad regression coverage.

Sequence diagram for safe manifest transformation and code generation

sequenceDiagram
    participant Manifest as Manifest
    participant Transformer
    participant Jinja
    participant Commander
    participant GeneratedCLI

    Manifest->>Transformer: load_manifest()
    Transformer->>Jinja: render manifest with env mapping
    Jinja-->>Transformer: rendered manifest data
    Transformer->>Commander: generate_cli()
    Commander->>Commander: py_str() and to_identifier()
    Commander-->>Transformer: escaped generated source
    Transformer->>GeneratedCLI: import generated source
Loading

Sequence diagram for collision-safe CLI loading and unloading

sequenceDiagram
    participant User
    participant Loader
    participant Bin as InterpreterBin
    participant CLIFile as CliffyCLIFile

    User->>Loader: load_from_cli(cli)
    Loader->>Bin: check target with lexists()
    alt foreign executable exists
        Bin-->>Loader: existing non-Cliffy file
        Loader-->>User: exit_err()
    else target is absent or Cliffy-owned
        Loader->>Bin: write entry script
        Loader->>CLIFile: write generated module
    end
    User->>Loader: unload_cli(cli_name)
    Loader->>Bin: is_cli_script()
    alt Cliffy entry script
        Loader->>Bin: remove script
    else foreign executable
        Loader-->>Bin: keep file
    end
Loading

Flow diagram for reliable CLI command outcomes

flowchart TD
    A[CLI command invoked] --> B{Operation succeeds?}
    B -->|Yes| C[Print success]
    B -->|No| D[Print error]
    D --> E[Exit with status 1]
    C --> F[Exit with status 0]
Loading

File-Level Changes

Change Details Files
Harden manifest transformation and generated Python against data injection and unsafe names.
  • Keep environment values as data while rendering manifests and validate CLI filenames.
  • Escape manifest-derived strings in Typer and Click source generation.
  • Normalize command, group, alias, and parameter names into safe identifiers, while applying global parameters consistently.
  • Fix parsing of defaults, aliases, full Typer options, greedy commands, and nested groups.
cliffy/helper.py
cliffy/manifest.py
cliffy/transformer.py
cliffy/commander.py
cliffy/parser.py
cliffy/commanders/click.py
cliffy/commanders/typer.py
tests/test_codegen.py
tests/test_transformer.py
Prevent destructive loader and metadata behavior when managing installed CLIs.
  • Reject collisions with foreign executables and only remove entry scripts recognized as Cliffy-owned.
  • Resolve loaded module paths correctly for dashed names.
  • Match metadata names literally, tolerate non-empty metadata directories, and close metadata files.
  • Remove stale generated modules and clean temporary files on failures.
cliffy/loader.py
cliffy/homer.py
cliffy/builder.py
cliffy/cli.py
tests/test_loader.py
tests/test_homer.py
tests/test_builder.py
tests/test_cli.py
Make dependency, include, encoding, documentation, and reload handling deterministic and robust.
  • Validate PEP 508 requirements using installed distribution metadata and normalize package names.
  • Resolve includes relative to the source manifest, preserve order, and report cycles or malformed manifests.
  • Use UTF-8 for manifest and generated-file I/O and detect exact watched paths.
  • Escape generated HTML, handle empty example descriptions, preserve reusable fallback table styles, and sort generated imports.
cliffy/helper.py
cliffy/transformer.py
cliffy/reloader.py
cliffy/doc.py
cliffy/rich.py
cliffy/commander.py
cliffy/commanders/click.py
cliffy/commanders/typer.py
tests/test_helper.py
tests/test_transformer.py
tests/test_encoding.py
tests/test_reloader.py
tests/test_doc.py
tests/test_rich.py
tests/test_codegen.py
Correct CLI test execution and command failure signaling.
  • Use Click's runner for Click-generated manifests and return exit code 1 for failed tests.
  • Propagate failures from validate, build, update, remove, and docs commands.
  • Bundle the framework imported by generated code when building zipapps.
cliffy/cli.py
cliffy/tester.py
cliffy/builder.py
tests/test_tester.py
tests/test_cli.py
tests/test_builder.py

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@coderabbitai

coderabbitai Bot commented Oct 12, 2026

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration
  • Configuration used: Repository UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: fbbfc288-b120-40e6-836d-5e91e3201096

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've found 8 issues

Fixed security issues:

  • Command injection from untrusted input passed to OS command execution (link)
Prompt for AI Agents
Please address the comments from this code review:

## Individual Comments

### Comment 1
<location path="cliffy/homer.py" line_range="103" />
<code_context>
         Optional[str]: CLI metadata path
     """
-    if cli_metadata_path := glob.glob(f"{CLIFFY_METADATA_DIR}/*/{cli_name}.json"):
+    if cli_metadata_path := glob.glob(f"{glob.escape(str(CLIFFY_METADATA_DIR))}/*/{glob.escape(cli_name)}.json"):
         return cli_metadata_path[0]
     return None
</code_context>
<issue_to_address>
**CLI removal escapes managed directories**

When a caller supplies a CLI name containing traversal components and a matching `.json` file exists at the resulting outside path, `glob.escape` does not escape path separators or `..`, so a crafted name can make `get_metadata_path` find a matching JSON file outside the metadata directory. `remove` then calls `remove_metadata` and `Loader.unload_cli` with that name, deleting files outside Cliffy's managed directories.

Validate CLI names at CLI entry points and perform removal using the exact metadata record's validated name and paths rather than reconstructing paths from raw arguments.

Also at `cliffy/homer.py:49`, `cliffy/loader.py:37`.
</issue_to_address>

### Comment 2
<location path="cliffy/loader.py" line_range="46-50" />
<code_context>
             os.remove(cls.get_cli_path(cli_name))

+    @staticmethod
+    def is_cli_script(script_path: str) -> bool:
+        """Whether the file is an entry script written by `deploy_script`"""
+        try:
+            with open(script_path, encoding="utf-8") as script:
+                return "from cliffy.clis." in script.read(4096)
+        except (OSError, UnicodeDecodeError):
+            return False
</code_context>
<issue_to_address>
**Foreign executables are overwritten**

When a foreign target or symlink target contains `from cliffy.clis.` in its first 4096 characters, `Loader.is_cli_script` treats the marker substring as proof of ownership, so `deploy_script` overwrites the target and `unload_cli` can delete it; the caller loses a foreign executable.

Verify launcher ownership using a reliable check that rejects foreign files and symlinks instead of accepting a marker substring.
</issue_to_address>

### Comment 3
<location path="cliffy/loader.py" line_range="21" />
<code_context>

     def deploy_script(self) -> str:
         script_path = Loader.get_cli_script_path(self.cli.name)
+        if os.path.lexists(script_path) and not Loader.is_cli_script(script_path):
+            # i.e. a manifest named `python` or `pip`: the script would replace that executable
+            exit_err(
</code_context>
<issue_to_address>
**Ownership checks race with writes**

When another process replaces the launcher path after the ownership check and before the write or removal, `deploy_script` or `unload_cli` uses the path after checking it, so a replacement executable or symlink target is overwritten or deleted.

Make launcher ownership checks and writes or removals race-safe, and do not follow symlinks.

Also at `cliffy/loader.py:39-41`.
</issue_to_address>

### Comment 4
<location path="cliffy/loader.py" line_range="38-41" />
<code_context>

     @classmethod
     def unload_cli(cls, cli_name: str) -> None:
+        script_path = cls.get_cli_script_path(cli_name)
+        if cls.is_cli_script(script_path):
+            with contextlib.suppress(FileNotFoundError):
+                os.remove(script_path)
         with contextlib.suppress(FileNotFoundError):
-            os.remove(cls.get_cli_script_path(cli_name))
</code_context>
<issue_to_address>
**Launcher is left unable to run**

When an existing launcher is edited or cannot be read during unload, `unload_cli` skips removing the launcher but still deletes its module; `remove` can also delete the metadata before this check. The remaining launcher cannot import its module, and the metadata no longer provides a normal way to clean it up.

Only delete the module and metadata after the launcher has been safely removed, or otherwise keep the remaining launcher runnable and manageable.
</issue_to_address>

### Comment 5
<location path="cliffy/builder.py" line_range="86" />
<code_context>
+                interpreter=interpreter,
+            )
+        finally:
+            delete_temp_files()

-    delete_temp_files()
</code_context>
<issue_to_address>
**Concurrent runs delete each other's files**

When two `run_cli` calls overlap while one is writing or importing its temporary script, `delete_temp_files` closes and unlinks every file in the process-wide `TEMP_FILES` list, so one run's cleanup removes another run's script and its import or use fails.

Track and clean up only the temporary files belonging to the current run.

Also at `cliffy/builder.py:34-35`, `cliffy/tester.py:52-54`.
</issue_to_address>

### Comment 6
<location path="cliffy/reloader.py" line_range="28" />
<code_context>

     def on_modified(self, event: FileSystemEvent) -> None:
-        if event.is_directory or not str(event.src_path).endswith(self.manifest_path):
+        # compared as full paths: `endswith` also matched other-x.yaml when watching x.yaml
+        if event.is_directory or os.path.realpath(os.fsdecode(event.src_path)) != self.manifest_realpath:
             return
</code_context>
<issue_to_address>
**Older reload overwrites newer state**

When a reload of an older manifest state runs slowly while a later edit is reloaded and committed first, `on_modified()` starts an independent reload thread for each eligible event. An older thread can finish after a newer one and overwrite the newer generated module and metadata, leaving the loaded CLI at an older manifest version.

Serialize reloads or attach a generation/version to each reload and prevent an older generation from committing after a newer one.
</issue_to_address>

### Comment 7
<location path="tests/test_codegen.py" line_range="214" />
<code_context>
+            "['a', 'b']",
+            id="builtin_command",
+        ),
+        pytest.param(BASE + "commands:\n  print: print('printed')\n", ["--version"], "t, 1", id="print_command"),
+        pytest.param(BASE + "commands:\n  my-group.sub: print(1)\n", ["my-group", "sub"], "1", id="dash_group"),
+        pytest.param(BASE + "commands:\n  2fa.setup: print(1)\n", ["2fa", "setup"], "1", id="digit_group"),
</code_context>
<issue_to_address>
**Builtin command test skips command**

When the generated CLI's version callback works but its `print` command is missing or broken, `test_names_become_valid_identifiers` uses the `print_command` case at line 214 with `args=["--version"]`, so it never dispatches the generated `print` command; the version output still passes if that command is missing or broken.

Invoke the `print` command in this case and assert its command output.
</issue_to_address>

### Comment 8
<location path="tests/test_loader.py" line_range="27" />
<code_context>

 if __name__ == "__main__":
     main()
</code_context>
<issue_to_address>
**Symlink setup fails on Windows**

When the test runs on Windows without Developer Mode or symlink-creation privileges, `test_load_does_not_overwrite_foreign_executable` reaches `Path.symlink_to` at line 26 before calling `Loader`; on Windows without symlink privileges, that raises `OSError` and fails the test even when the loader is correct.

Skip this test when creating the symlink is unsupported, while keeping the symlink-based assertions on systems that permit it.
</issue_to_address>

Sourcery assessment

Needs a human reviewer. 8 findings to address first, and if an unrelated executable in the bin directory contains from cliffy.clis. in its first 4096 bytes, the ownership check can treat it as Cliffy’s and overwrite it during load or delete it during unload. That file loss persists after reverting and requires restoring the executable separately.

Blocking findings: cliffy/homer.py:103, cliffy/loader.py:50, cliffy/loader.py:21, cliffy/loader.py:41, cliffy/builder.py:86, and 3 more


Sourcery is free for open source - if you like our reviews please consider sharing them ✨

Comment thread cliffy/homer.py Resolved
Comment thread cliffy/loader.py Resolved
Comment thread cliffy/loader.py

def deploy_script(self) -> str:
script_path = Loader.get_cli_script_path(self.cli.name)
if os.path.lexists(script_path) and not Loader.is_cli_script(script_path):

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Critical · Ownership checks race with writes

When another process replaces the launcher path after the ownership check and before the write or removal, deploy_script or unload_cli uses the path after checking it, so a replacement executable or symlink target is overwritten or deleted.

Make launcher ownership checks and writes or removals race-safe, and do not follow symlinks.

Also at cliffy/loader.py:39-41.

Prompt for AI agents
In `cliffy/loader.py` at line 21:

**Ownership checks race with writes**

When another process replaces the launcher path after the ownership check and before the write or removal, `deploy_script` or `unload_cli` uses the path after checking it, so a replacement executable or symlink target is overwritten or deleted.

Make launcher ownership checks and writes or removals race-safe, and do not follow symlinks.

Also at `cliffy/loader.py:39-41`.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mostly fixed in #60 (1459ad2): launchers and modules are written to a temp file in the same directory and renamed, a new launcher is created exclusively, reads use lstat and O_NOFOLLOW, and the ownership check runs right before unlink. A window remains: replacing or deleting an existing cliffy launcher is still check-then-rename/unlink, so a file another process puts at that exact path in between is replaced or removed. Links are never followed, so nothing outside the bin directory is touched. This is recorded in known-issues.

Comment thread cliffy/loader.py Resolved
Comment thread cliffy/builder.py Resolved
Comment thread cliffy/reloader.py Resolved
Comment thread tests/test_codegen.py Resolved
Comment thread tests/test_loader.py Resolved
@jaykv
jaykv changed the base branch from chore/python-3.11-3.15 to main October 12, 2026 21:22
@jaykv
jaykv merged commit f3ce576 into main Oct 12, 2026
7 checks passed
@jaykv
jaykv deleted the fix/audit-bugs-and-security branch October 12, 2026 21:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant