Repository navigation
fix: 실제 Azure 배포 경로 보완 - #2
Merged
Merged
Conversation
Apply fixes verified through a live Foundry deployment: execute the custom Bicep infrastructure, support uv-backed hooks, provision Search and toolboxes idempotently, resolve hosted-agent identities for RBAC, and feed exported Foundry evaluation results into all three deployment gates. Preserve Build -> Evaluate -> Operate ordering so evaluator errors block continuous evaluation and day-two controls. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Remove system Python directories from the fallback contract PATH so Linux and macOS both exercise the uv launcher branch deterministically. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Improves live Azure deployment reliability while preserving the Build → Evaluate → Operate sequence.
Changes:
- Corrects Bicep provisioning, Search naming/region, hooks, RBAC, and Toolbox validation.
- Aligns evaluations with live Foundry CLI formats and blocks evaluator errors.
- Expands deployment contract tests and documentation.
Reviewed changes
Copilot reviewed 25 out of 25 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
.github/workflows/deploy-evaluate.yml |
Runs three agent evaluations and gates results. |
azure.yaml |
Switches infrastructure to Bicep. |
deploy/hooks/postdeploy.ps1 |
Adds Python fallback selection. |
deploy/hooks/postdeploy.sh |
Adds Python fallback selection. |
deploy/hooks/postprovision.ps1 |
Adds Python fallback selection. |
deploy/hooks/postprovision.sh |
Adds Python fallback selection. |
deploy/infra/main.bicep |
Separates Search region and suffixes names. |
deploy/infra/main.bicepparam |
Reads deployment parameters from azd environment. |
evals/eval.yaml |
Adds development live-evaluation configuration. |
evals/human-resources.yaml |
Adds HR live-evaluation configuration. |
evals/marketing.yaml |
Adds marketing live-evaluation configuration. |
README.md |
Updates deployment and evaluation instructions. |
src/lifecycle_ops/evaluation/gate.py |
Parses live evaluator counts and errors. |
src/lifecycle_ops/provisioning/knowledge_bases.py |
Fixes Search payload and error reporting. |
src/lifecycle_ops/provisioning/rbac.py |
Resolves agent identities through azd. |
src/lifecycle_ops/provisioning/toolboxes.py |
Validates connection ARM properties. |
tests/ops/test_agent_rbac_matrix.py |
Tests azd identity extraction. |
tests/ops/test_knowledge_manifest.py |
Tests Search payload and error details. |
tests/ops/test_toolbox_ops.py |
Tests ARM connection validation. |
tests/ops/test_validate_eval_results.py |
Tests evaluator-error gating. |
tests/repo/test_documentation.py |
Updates README command contracts. |
tests/repo/test_eval_config_contract.py |
Validates per-agent evaluation configs. |
tests/repo/test_hooks_contract.py |
Tests Python fallback behavior. |
tests/repo/test_infra_static.py |
Tests Bicep provider and Search settings. |
tests/repo/test_workflows.py |
Tests multi-agent evaluation workflow. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Use DefaultAzureCredential for ARM connection reads, restore shared and cross-department cases in each live evaluation dataset, and document safe cleanup of legacy fixed-name Search services after migration verification. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 28 out of 28 changed files in this pull request and generated 2 comments.
Suppressed comments (1)
deploy/infra/main.bicepparam:4
AZURE_SEARCH_LOCATIONis configurable locally, but the deployment workflow maps and persists onlyAZURE_LOCATION; on a clean runner this parameter always falls back tocentralus. That prevents operators from redirecting Search when Central US has capacity or policy constraints. Map a workflow variable forAZURE_SEARCH_LOCATIONand set it in the selected azd environment before provisioning.
param searchLocation = readEnvironmentVariable('AZURE_SEARCH_LOCATION', 'centralus')
Closed
daeungo1
added a commit
that referenced
this pull request
Aug 17, 2026
PR #2 (refactor/lifecycle-structure) and this branch fixed the same live deployment independently. Both are kept; where they overlap the stronger implementation wins. From PR #2: - azure.yaml switches infra.provider from microsoft.foundry to bicep and main.bicepparam reads the azd environment, so deploy/infra is finally deployed. This is the root fix for the behaviour this branch had to work around: the foundry provider synthesised its own template and silently dropped the Search services, role assignments and everything else declared in main.bicep. - Search services get a deterministic name suffix and their own AZURE_SEARCH_LOCATION, defaulting to Central US where capacity is available. - Hooks resolve an interpreter through PYTHON, python, uv and python3 instead of assuming `python` exists. - Agent principal ids come from `azd ai agent show`, which avoids the azure-ai-projects version differences this branch had to work around. - The gate blocks on live evaluator errors and strips the builtin. prefix when matching metric names. - Per-agent eval configs with single JSONL datasets. From this branch: - Application Insights, the Foundry AppInsights connection and per-agent OTEL_SERVICE_NAME, so all three agents are traceable in Foundry. - Case-insensitive azd environment lookups, needed on Linux runners. - azure/login@v2 in the deploy workflow for the RBAC hook. - shutil.which resolution so the Azure CLI can be spawned on Windows. builtin.tool_call_accuracy is removed from all three eval configs. The evaluator requires a tool_definitions input the CLI cannot supply, so every sample errors, and PR #2's gate now blocks on evaluator errors — keeping it would make promotion permanently red for a reason unrelated to agent quality. The rationale and the re-enable condition are recorded in evals/eval.yaml and docs/operations.md. The Windows uv-fallback test now installs a batch shim instead of a shebang script, and the POSIX variant skips on Windows, so the suite is green on both. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
daeungo1
added a commit
that referenced
this pull request
Aug 17, 2026
origin/main gained the merged PR #2 plus a follow-up that moved the live evaluation datasets to evals/live/ and read ARM connections with DefaultAzureCredential. Those land here; the live datasets are rewritten in Korean for the fictional enterprise scenario while keeping the upstream contract of eight rows per department, six allow cases including one shared-handbook citation, and two cross-department denials. builtin.tool_call_accuracy stays out of all three configs: the evaluator needs a tool_definitions input the CLI cannot supply, so every sample errors and the gate, which blocks on evaluator errors, would never pass. README now carries three purpose-built SVG views instead of Mermaid: the lifecycle stages, the three-department scenario, and the Azure resources. A contract test estimates label widths per character class, since Hangul and Latin advance very differently, and fails if any label leaves its box or the canvas. The scenario is described as fictional throughout rather than naming a company. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
배경
PR #1 병합 이후 실제 Azure 리소스 생성과 Hosted Agent 호출까지 수행하면서 정적 테스트에서 발견되지 않은 배포 결함을 확인했습니다. 이 PR은 실제 환경에서 재현·수정·재검증한 후속 변경만 포함합니다.
주요 변경
microsoft.foundryprovider가deploy/infra사용자 Bicep을 배포하지 않음bicepprovider 적용, azd 환경값을 Bicep 매개변수에 연결AZURE_SEARCH_LOCATION분리, 기본값 Central USpython명령 부재로 종료 코드 127PYTHON→python→ uv Python 3.13 →python3실행기 선택azd ai agent show기반 principal ID 조회builtin.*ID, 단일 JSONL,eval show --out-file적용per_testing_criteria_resultspass rate 계산과 evaluator error 차단실제 Azure 검증
gpt-5.4-mini: East US 2response.completedazd provision·azd deploy재실행 성공Evaluate 결과
Evaluate 실패 후 continuous evaluation과 Agent 365 제어가 실행되지 않아 Build → Evaluate → Operate 차단 순서를 실제로 확인했습니다.
로컬 검증
운영 주의