Repository navigation
Phase 3: Eval & trust — TR suites, score reasons, weighted intent, CI badge - #11
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 79915bb18b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| run: | | ||
| mkdir -p docs/benchmarks | ||
| promptfix eval --report docs/benchmarks/report-$(date +%Y%m%d).html | ||
| python -m promptfix eval --report eval-report.html 2>&1 | tee eval-output.txt || true |
There was a problem hiding this comment.
This step now runs python -m promptfix eval, but the package has no promptfix/__main__.py (the CLI is exposed via the promptfix console script in pyproject.toml), so this invocation exits with No module named promptfix.__main__ instead of executing the eval suite. As written, the workflow cannot generate real eval output or reports from this command.
Useful? React with 👍 / 👎.
| run: | | ||
| mkdir -p docs/benchmarks | ||
| promptfix eval --report docs/benchmarks/report-$(date +%Y%m%d).html | ||
| python -m promptfix eval --report eval-report.html 2>&1 | tee eval-output.txt || true |
There was a problem hiding this comment.
Appending || true makes this step succeed even when the eval command crashes or returns a failure status, so regressions and broken CI setup are silently ignored. In this workflow, that means scheduled/manual runs can look green while no valid evaluation actually ran, which undermines the trust signal from the report job.
Useful? React with 👍 / 👎.
Phase 3 — Evaluation & Trust
All 341 tests green, ruff + mypy clean.