ci: remove Azure Artifacts publishing - #11
Merged
Conversation
Drop the private UiPath Azure Artifacts (`coder_eval`) feed as a publish target. Delete publish-dev.yml (its sole purpose was publishing per-PR dev wheels to that feed) and remove the "Publish to coder_eval feed" twine step from release.yml, keeping public PyPI, GHCR, and the git tag intact. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Claude finished @akshaylive's task in 1m 24s —— View job Code Review In Progress
|
bai-uipath
approved these changes
Jul 9, 2026
uipreliga
reviewed
Jul 9, 2026
uipreliga
left a comment
uipreliga
left a comment
Collaborator
There was a problem hiding this comment.
Review: coder_eval — pr:11
Scope: pr:11 · branch pr-11 (akshaya/remove_azure_publish) · 2082dcc · 2026-07-09T22:31Z · workflow variant
Change class: simple — removes the Azure Artifacts publish step from release.yml and deletes publish-dev.yml; no job dependencies or control flow affected, the PyPI + GHCR release path is unchanged.
coder_eval is in excellent health — all eight axes score 9.9–10 with zero critical/high/medium findings and no defect that can change a task's score or final_status for identical agent output — so the only real risk is documentation drift (one stale CI comment), and the bottom line is that the codebase is ship-ready with cosmetic cleanup only.
Summary
| Axis | Score | 🔴 | 🟠 | 🟡 | 🔵 | Top Issue |
|---|---|---|---|---|---|---|
| 1. Code Quality & Style | 9.9 / 10 | 0 | 0 | 0 | 1 | Stale comment: runner-choice justification still lists twine, but the PR removed the only twine step |
| 2. Type Safety | 10 / 10 | 0 | 0 | 0 | 0 | — |
| 3. Test Health | 10 / 10 | 0 | 0 | 0 | 0 | — |
| 4. Security | 10 / 10 | 0 | 0 | 0 | 0 | — |
| 5. Architecture & Design | 10 / 10 | 0 | 0 | 0 | 0 | — |
| 6. Error Handling & Resilience | 10 / 10 | 0 | 0 | 0 | 0 | — |
| 7. API Surface & Maintainability | 10 / 10 | 0 | 0 | 0 | 0 | — |
| 8. Evaluation Harness Quality | 10 / 10 | 0 | 0 | 0 | 0 | — |
Overall Score: 10 / 10 · Weakest Axis: Code Quality & Style at 9.9 / 10
Totals: 🔴 0 · 🟠 0 · 🟡 0 · 🔵 1 across 8 axes.
Blockers
None.
Non-blocking, but please consider before merge
None.
Nits
- [Axis 1] Stale comment: runner-choice justification still lists
twine, but the PR removed the only twine step (.github/workflows/release.yml:35) — Line 35 reads# \uipath-ubuntu-latest` pool. uv, twine, and the docker buildx -> GHCR— it justifies the GitHub-hostedruns-on: ubuntu-latestchoice by listing the tools that "run fine here." This PR deleted the only twine invocation in the workflow (the removedPublish to coder_eval feedstep, which ranuv tool run --from twine twine upload ...). The retained PyPI publish now uses thepypa/gh-action-pypi-publishaction (line 162), anduv build(line 82) builds — notwineis invoked anywhere in the final file (grep confirms line 35 is the soletwineoccurrence). The removal thus left a half-removed reference: droptwinefrom the comment so it matches the code (e.g.# uv and the docker buildx -> GHCR push all run fine here.`). Cosmetic only; no effect on execution.
What's Missing
Daily/nightly:
- 🟠 Unstated daily/nightly blast radius: the release job's Azure
coder_evalfeed publish (release.yml) and the per-PR dev-wheel publish (publish-dev.yml) were the private UiPath distribution channel, but the nightly/eval-runner pipeline lives in the separate coder-eval-uipath repo and this PR does not state whether that pipeline installs coder-eval from the Azure feed. If it does, released versions now land only on public PyPI and the nightly's internal index goes stale, breaking it under version skew — the cross-repo contract change is unaddressed. (trigger: .github/workflows/release.yml)
Downstream consumers:
- 🟡 No migration note for consumers pinned to the Azure feed: any UiPath-internal consumer with uv/pip index config pointing at pkgs.dev.azure.com/uipath/ML%20Platform/_packaging/coder_eval/pypi/simple/ stops resolving new releases, and deleting publish-dev.yml removes the
build:devper-PR dev-wheel channel authors used to hand an installable pre-release wheel to downstream repos for integration smoke-tests. Public PyPI (and the TestPyPI dry-run job) are viable alternatives, but the PR points no consumer at them. (trigger: .github/workflows/publish-dev.yml)
Harness & Lint Improvements
Nothing identified.
Top 5 Priority Actions
- No scoring/final_status-affecting defects were found, so no fix is required to protect determinism — the harness produces identical scores for identical agent output.
- Fix the stale runner-choice comment at .github/workflows/release.yml:35 by dropping the now-removed
twinereference (the PR deleted the only twine step; publishing now goes through pypa/gh-action-pypi-publish at line 162 anduv buildat line 82). - Add a lightweight CI/lint guard that flags references to removed tools in workflow comments so half-removed mentions like this twine reference are caught mechanically rather than in review, per CLAUDE.md's 'could a lint rule have prevented this?' guidance.
- Sweep the remaining GitHub workflow comments once for other post-refactor drift (tool names, step names, pool justifications) while release.yml is already open, since this class of staleness is cosmetic but recurs after CI restructures.
- Keep the current bar in place — treat the 10/10 result as a baseline and re-run the 8-axis review on the next substantive change rather than adding speculative hardening, consistent with the project's YAGNI/KISS principles.
Stats: 0 🔴 · 0 🟠 · 0 🟡 · 1 🔵 across 8 axes reviewed.
CarlesUIPath
added a commit
that referenced
this pull request
Jul 28, 2026
…ly path, -O safe) resolve_route is reached on the evaluate-only path without a preceding validate_api_keys(), so a scheme-less/missing LITELLM_BASE_URL there escaped validation and recorded an empty host in environment_info; the arm's asserts were also strippable under 'python -O'. Call _validate_litellm_settings() in the LiteLLM arm (raise, not assert) so both the normal and evaluate-only paths validate presence + URL scheme with a field-named error; narrowing asserts kept for pyright only. Closes the blocker-2 evaluate-only edge + review non-blocking #11. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
CarlesUIPath
added a commit
that referenced
this pull request
Jul 28, 2026
…laude Code harness (#54) * feat(routing): add LiteLLM backend — skill activation works on open-weight models Adds a LiteLLMRoute (ApiBackend.LITELLM) so the Claude Code SDK can drive EU-resident Bedrock open-weight models (GLM 5, DeepSeek V3.2) through a self-hosted LiteLLM proxy that translates Anthropic Messages <-> Bedrock Converse. Because the model runs inside the native Claude Code harness, the Skill tool and progressive disclosure fire unchanged — so UiPath skill activation works on a non-Claude model via this path. Verified end-to-end: the LiteLLM integration works AND drives skill activations. Tested on a real uipath maestro_flow skill task (cli_dice_roller) using GLM 5 — 4/4 criteria passed (api_routing=litellm, model=zai.glm-5). The agent engaged the uipath-maestro-flow skill and used `uip maestro flow node add` / `edge add` / `validate` correctly, producing a schema-valid flow. - ApiBackend.LITELLM + LiteLLMRoute frozen dataclass in the ApiRoute union - litellm_* settings + fail-fast _validate_litellm_settings (config.py) - SDK env wiring in ClaudeCodeAgent (_build_sdk_env / _resolve_effective_model): sets ANTHROPIC_BASE_URL/AUTH_TOKEN/MODEL, neutralizes inherited ANTHROPIC_API_KEY - docker/litellm-config.yaml: GLM 5 / DeepSeek V3.2 -> Bedrock Converse @ eu-north-1 - tests/test_litellm_route.py Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(routing): --backend litellm, open-weight pricing, and correct cost accounting Observability + CLI: - `--backend litellm` flag (alias for API_BACKEND=litellm) - Record the LiteLLM route in run artifacts (host + model only — never the auth token or full base_url) and in the routing log line - Guardrail test: every ApiRoute member must be handled at each route-matching seam (ROUTE_NAMES / _build_sdk_env / _format_routing) Pricing + cost: - Add rates for zai.glm-5 ($1.55/$4.96) and deepseek.v3.2 ($0.74/$2.22) - _normalize_model strips LiteLLM/Bedrock routing prefixes (converse/, bedrock/converse/) so SDK-reported model ids resolve to a rate - On the litellm backend, recompute total_cost_usd from the token buckets at the model's real rate: the Claude SDK's costUSD assumes Claude pricing and is wrong for open-weight models. Token buckets are untouched (the per-message reconciliation invariant is preserved); an unpriced model yields None (an honest N/A) instead of the misleading SDK figure. Also drops internal plan-phase references from the shared LiteLLM config comments. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(routing): log the effective litellm model + cover LITELLM in the enum test - _format_routing now shows the effective agent model (e.g. from --model), not the route's LITELLM_MODEL default, so the 'API routing:' line reflects what the agent actually sends. - Update test_api_backend_enum_values to include ApiBackend.LITELLM (ripple from adding the enum member). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(litellm): add Kimi K2.5, correct GLM eu-north-1 rate, drop Qwen; add proxy start script - config: add moonshotai.kimi-k2.5 to the shared litellm model_list; remove qwen.qwen3-coder-next (needs a Bedrock model-access grant — deferred) - pricing: Kimi K2.5 ($0.72/$3.60); correct GLM 5 (zai.glm-5) to the eu-north-1 rate ($1.20/$3.84, was the eu-west $1.55/$4.96) - docker/start-litellm.sh: launch the proxy with the required creds exported. Reads AWS_BEARER_TOKEN_BEDROCK / AWS_REGION out of .env and exports them into the proxy's process (the bare-KEY=value .env is otherwise unexported → "Unable to locate credentials" 401), sets LITELLM_MASTER_KEY, fails loud on a missing token, and kills any stale listener before relaunching. Pure-bash value extraction (handles quoted values + inline comments; no BSD-sed quirks). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(litellm): preflight external proxy reachability (fail fast, no hang) Ping <LITELLM_BASE_URL>/health/liveliness once before the batch when the litellm backend targets an external proxy. If unreachable, exit with a clear error instead of letting every task hang on the dead endpoint. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(docker): forward LiteLLM env into containers, rewrite loopback proxy URL Docker-driver tasks on the litellm backend failed because the LITELLM_* creds weren't in the env allowlist: the in-container Settings saw API_BACKEND=litellm with no creds and validation raised. Add LITELLM_BASE_URL/AUTH_TOKEN/MODEL/SMALL_MODEL to the default env_passthrough allowlist. LITELLM_BASE_URL is special-cased: a loopback host is rewritten to host.docker.internal (the proxy runs on the host, not in the container) and published via --add-host for Linux parity; the token stays name-only so it's never rendered in the logged argv. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(litellm): disable Claude Code attribution metadata (Bedrock requestMetadata 400) * fix(litellm): neutralize inherited Bedrock creds so the CLI uses the proxy, not Bedrock-direct * feat(pricing): add OpenRouter open-weight models (Kimi K3, GLM 5.2, DeepSeek V4 Pro) Wire three OpenRouter-hosted open-weight models through the existing litellm backend (cost-optimization path) and register their OpenRouter-published rates. These providers cache prompt prefixes implicitly, so cache-creation stays 0 and only cache-read carries a (discounted) rate. Also corrects the Kimi K3 cache-read rate from $0 to the published $0.30/M. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(litellm): pin OpenRouter provider by price for deterministic cost OpenRouter routes each request to a provider chosen at request time, which can be pricier than the model page's headline. Pin extra_body.provider={sort: price, allow_fallbacks: false} on the OpenRouter models so the cheapest upstream is used deterministically and the billed rate matches the pricing table. Also reindents an evaluate-only logging call. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(evalboard): mirror OpenRouter model rates into pricing table Add Kimi K3, GLM 5.2, DeepSeek V4 Pro to evalboard's ported pricing table so the per-stage / per-task cost column populates for litellm-backend runs (was blank — the models were absent). Rates mirror src/coder_eval/pricing.py; accurate only with provider pinning (sort: price) in the LiteLLM config. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(evalboard): price the reconciliation row so the Cost column sums to the turn total The synthetic reconciliation row (tokens billed but not streamed per-message) was rendered with a hardcoded '—' cost and built with costUsd=null. For providers whose per-message stream is sparse (OpenRouter/LiteLLM), that row holds most of the turn's input+cache tokens, so summing the visible Cost column fell far short of the real total. Now the reconciliation entry is priced from the turn's model (added model_used to TurnEntry) and its row renders that cost. Safe from double-counting: the cost simulator excludes reconciliation by role, and the authoritative task total reads the backend aggregate, not a sum of per-message costUsd. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(evalboard): relax pricing-parity guard to subset-match Exact key-set parity forced lib/pricing.ts to mirror every backend model, even ones the evalboard never renders — so unrelated backend additions had to be copied here just to keep the build green. Relax to subset semantics: every model priced in lib/pricing.ts must exist in pricing.py with identical rates (catches a wrong or orphan frontend rate), without requiring the frontend to carry models it doesn't display. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(litellm): export OPENROUTER_API_KEY into the proxy env The proxy's openrouter/* models authenticate with OPENROUTER_API_KEY, which the proxy reads from its own environment. Read it out of .env alongside the Bedrock creds so the OpenRouter models work end-to-end without a manual export. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * style(tests): ruff-format LiteLLMRoute test calls (collapse to one line) Pure formatting: ruff at line-length 120 collapses two multi-line _build_sdk_env(LiteLLMRoute(...)) calls onto single lines. No logic change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(litellm): add docker/LITELLM.md — proxy setup + how to add models Explains when to run start-litellm.sh, the litellm-config.yaml structure, and the add-a-model recipe (Bedrock + OpenRouter, incl. mandatory pricing.py/evalboard pricing entries, proxy restart, and the OpenRouter provider-routing cost caveat). Adds a pointer to it from the config header. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor(litellm): move proxy config/script/docs to top-level litellm/ The LiteLLM proxy is a first-class, long-lived component (launched via uvx, not Docker), so it lives in its own top-level litellm/ dir rather than under docker/. Moves litellm-config.yaml + start-litellm.sh + LITELLM.md->README.md and updates all path references (start-script CONFIG default, run_command preflight message, config header, README). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(litellm): fix stale _build_sdk_env assertion for the LiteLLM route The custom-route env test asserted CLAUDE_CODE_USE_BEDROCK / AWS_BEARER_TOKEN_BEDROCK were absent, but the LiteLLM route deliberately blanks them to "" to neutralize inherited Bedrock creds (adfefd3) so the CLI can't bypass the proxy. Assert they are present-and-empty instead. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(litellm): silence bandit B310 on the preflight urlopen The litellm preflight opens LITELLM_BASE_URL/health/liveliness to fail fast when the proxy is down. bandit flags urlopen (B310, permitted-schemes) at medium — but the URL is built from the operator-configured LITELLM_BASE_URL, not untrusted input, so scope a nosec B310 with justification. Unblocks the Quality Gate. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(routing): pin llm_judge + simulated user to a constant Claude eval route Under --backend litellm the run's LiteLLMRoute reached the judge and the simulated user: the judge dispatch had no arm for it (silent score=0.0 with a misleading 'no usable API route'), and the simulator ran on the open-weight model under test — both break comparability against the Claude baseline. Add resolve_evaluation_route(settings, agent_route): Bedrock/Direct agents reuse their route unchanged; a LiteLLM agent pins evaluation to Bedrock (from the AWS bearer token) or Direct (ANTHROPIC_API_KEY), else DirectRoute(None) so llm_judge returns its clean 'unconfigured' error rather than 0.0. The orchestrator wires eval_route to SuccessChecker (llm_judge/agent_judge) and UserSimulator while the agent keeps the LiteLLM route; eval_routing is recorded in environment_info. Make the llm_judge dispatch exhaustive (explicit LiteLLMRoute + None arms, no wildcard) so pyright flags future route members. Extend test_route_seam_exhaustiveness to the judge-dispatch and _record_route_environment_info seams, and add a wiring test that the simulator receives the eval route, not the agent route. Addresses PR #54 review blocker 1. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(config): validate LITELLM_BASE_URL scheme (clean error, honest nosec) A scheme-less LITELLM_BASE_URL (e.g. 'localhost:4000') made the preflight's urlopen raise a bare ValueError ('unknown url type') that escaped as a traceback instead of a clean exit, and left environment_info's urlparse hostname empty. _validate_litellm_settings now rejects a non-http(s) / host-less URL with a field-named error (surfaced via validate_api_keys), and the CLI preflight guards the scheme before urlopen, returning a clean message. The preflight guard also makes the '# nosec B310' honest — the scheme is now constrained to http(s), which is exactly what B310 audits. Addresses PR #54 review blocker 2 (+ nit: nosec B310 scheme constraint). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(evalboard): mirror Bedrock open-weight rates + strip routing prefixes in resolvePricing Cost column rendered '—' for Bedrock-routed litellm runs: the 3 Bedrock open-weight ids (deepseek.v3.2, zai.glm-5, moonshotai.kimi-k2.5) were priced in pricing.py but absent from pricing.ts, and resolvePricing lacked the bedrock/converse/ + region prefix stripping that _normalize_model has (the recorded model_used arrives as e.g. 'converse/zai.glm-5'). Add the three ids, mirror _normalize_model's prefix stripping in resolvePricing, and add a parity direction (pricing.py -> pricing.ts) with an explicit DELIBERATELY_UNMIRRORED allowlist so a future litellm-relevant omission breaks the build instead of silently rendering '—'. Addresses PR #54 review blocker 4. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(agents): test litellm reprice wiring; warn on rate-card miss; dedupe pricing The reprice wiring (finalize -> _reprice_for_litellm) was untested — inverting the guard left the suite green while every litellm turn silently kept the SDK's Claude-priced cost, and an unpriced model yielded total_cost_usd=None that made the orchestrator skip the max_usd gate with no diagnostic. Extract _finalize_token_usage on _ClaudeTurnState so the wiring is directly testable, and add tests asserting a priced model reprices to the rate-table figure and an unpriced one yields None. Give _reprice_for_litellm the rate-card-miss warning its sibling _backfill_cost has, make it return None (the result was discarded), and extract the shared _price_from_buckets helper. Addresses PR #54 review blocker 3. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(routing): validate LITELLM settings in resolve_route (evaluate-only path, -O safe) resolve_route is reached on the evaluate-only path without a preceding validate_api_keys(), so a scheme-less/missing LITELLM_BASE_URL there escaped validation and recorded an empty host in environment_info; the arm's asserts were also strippable under 'python -O'. Call _validate_litellm_settings() in the LiteLLM arm (raise, not assert) so both the normal and evaluate-only paths validate presence + URL scheme with a field-named error; narrowing asserts kept for pyright only. Closes the blocker-2 evaluate-only edge + review non-blocking #11. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Removes the private UiPath Azure Artifacts (
coder_eval) feed as a publish target..github/workflows/publish-dev.yml— its sole purpose was building per-PR dev wheels and uploading them to the Azure feed (via thebuild:devlabel). Nothing else referenced it..github/workflows/release.yml— removed thePublish to coder_eval feedtwine step and updated the stale Azure references in the header/job comments.Public PyPI publishing (OIDC Trusted Publishing), the GHCR agent image, and the
v<version>git tag are untouched.pr-checks.yml's use of the Azure feed as a dependency index (install source, not publish target) is intentionally left in place.🤖 Generated with Claude Code