sre-lab: harden guided scenario operations

[hellices]

Summary

• make alert fire/resolve polling resilient to transient and malformed Azure responses
• preserve safe recovery and explicit failed state across every abnormal scenario exit
• gate Storage RBAC injection on required outputs and data-plane propagation
• validate Agent role evidence before teardown and document copyable setup/recovery commands
• prevent successful captures from being downgraded while preserving render-only recovery

Review cycles

• Round 1: scenario correctness and safety — clean
• Round 2: guided-lab operability and teardown — clean
• Round 3: independent regression/privacy/compatibility — clean

Verification

• python3 -m pytest scripts infra -q — 587 passed
• app/.venv/bin/python -m pytest app -q — 10 passed
• Bash 3.2 syntax and Python compile checks passed
• all infra/*.bicep templates built successfully

Notes

No Azure resources were created for this review-only hardening pass.

[Copilot]

Pull request overview

Hardens the SRE Agent event lab’s guided scenario scripts and operator workflows, focusing on making scenario execution/capture/teardown more resilient to transient Azure failures and more explicit/safe on abnormal exits.

Changes:

• Improves run-scenario.sh resilience: distinguishes “alert never fired” from “could not query alerts”, captures error evidence, adds S3 RBAC data-plane propagation gating, and ensures aborted runs are recorded as failed.
• Prevents “successful capture” (conclusion) from being downgraded by later capture attempts; adds CLI support to query capture status.
• Strengthens operator guidance and safety checks around Agent setup evidence, RBAC cleanup evidence, and manual recovery commands; expands test coverage accordingly.

Reviewed changes

Copilot reviewed 19 out of 19 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
monitor/sre-agent-event-lab/scripts/tests/test_lab_state.py	Adds regression coverage for refusing to downgrade an already-successful capture conclusion.
monitor/sre-agent-event-lab/scripts/tests/test_lab_scripts.py	Adds extensive scenario-script behavior tests for transient alert failures, abort recording, invalid JSON handling, and S3 propagation gating.
monitor/sre-agent-event-lab/scripts/tests/test_lab_guides.py	Asserts new/updated guide content for alert budgets, role assignment setup, and S3 propagation + manual recovery.
monitor/sre-agent-event-lab/scripts/tests/test_doctor.py	Adds coverage for missing/invalid assignment evidence and case-insensitive ARM IDs.
monitor/sre-agent-event-lab/scripts/tests/test_common.py	Verifies default alert resolution budget updates are reflected in scripts.
monitor/sre-agent-event-lab/scripts/tests/test_cleanup_external.py	Adds teardown safety test for empty assignment IDs recorded with principals; strengthens guide references in errors/docs.
monitor/sre-agent-event-lab/scripts/tests/lab_script_harness.py	Extends the harness to simulate more failure modes (jq failures, loadgen failures, transient AZ errors, etc.).
monitor/sre-agent-event-lab/scripts/run-scenario.sh	Implements resilient alert polling, explicit failed recording on abnormal exits, and S3 RBAC propagation gating.
monitor/sre-agent-event-lab/scripts/query-evidence.sh	Gates evidence output directory creation on required deployment outputs being present.
monitor/sre-agent-event-lab/scripts/lab_state.py	Prevents downgrading a successful capture, and adds capture-status CLI command.
monitor/sre-agent-event-lab/scripts/doctor.sh	Improves actionable error messaging and validates recorded Monitoring Contributor assignment evidence.
monitor/sre-agent-event-lab/scripts/common.sh	Centralizes a lowercase() helper and updates default alert resolution budget.
monitor/sre-agent-event-lab/scripts/cleanup-external.sh	Uses shared lowercase() and improves safety/diagnostics for evidence validation and cleanup.
monitor/sre-agent-event-lab/scripts/capture-scenario.sh	Adds refusal to re-capture once a conclusion exists, improves evidence validation and retry guidance.
monitor/sre-agent-event-lab/README.md	Documents the “cancelled azd down” recovery step requiring evidence refresh with new assignment IDs.
monitor/sre-agent-event-lab/guides/05-results.md	Documents that a concluded capture cannot be collected again within the same run attempt.
monitor/sre-agent-event-lab/guides/04-scenario-s3.md	Documents the RBAC propagation wait, failure mode, and a copyable manual restore command.
monitor/sre-agent-event-lab/guides/02-scenario-s1.md	Updates alert resolution budget documentation (25 minutes) and explains stateful log alert behavior.
monitor/sre-agent-event-lab/guides/01-agent-setup.md	Adds explicit role creation commands, captures assignment IDs, and validates agent-setup.json completeness.

Suppressed comments (1)

monitor/sre-agent-event-lab/scripts/run-scenario.sh:222

• The EXIT trap currently constructs a generic run aborted with status ... reason whenever OUTCOME_RECORDED is still 0. If a specific failure reason was already known (but the earlier mark-failed write failed), this loses the more actionable reason and can later record the generic one instead. Use the previously captured reason when available.

  if [[ "${OUTCOME_RECORDED}" -eq 0 ]]; then
    failure_reason="run aborted with status ${original_status}"
    if [[ "${recovery_status}" -ne 0 ]]; then
      failure_reason="${failure_reason}; automatic recovery failed"
    fi
    if ! record_failed_run "${failure_reason}"; then
      echo "CRITICAL: could not record the aborted ${SCENARIO} run as failed." >&2
    fi

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated no new comments.

Suppressed comments (1)

monitor/sre-agent-event-lab/scripts/tests/lab_script_harness.py:367

• In the python3 stub, mark_failed_failures (the decrementing counter) is not honored, while the .venv/bin/python stub supports it. Because make_lab() supports venv_present=False, tests that rely on mark_failed_failures will behave differently (or silently stop simulating retries) when the virtualenv is absent.

  *lab_state.py)
    if [[ "$*" == *" mark-failed "* && -f "${{state}}/mark_failed_fails" ]]; then
      exit 3
    fi
    exec "{REAL_PYTHON}" "$@"
    ;;