Skip to content

fix(mwpw-184989): send x-session-id to the LLM proxy and stop Feature QA failing silently - #600

Merged
sanrai merged 2 commits into
mainfrom
fix/qa-llm-session-id
Aug 27, 2026
Merged

fix(mwpw-184989): send x-session-id to the LLM proxy and stop Feature QA failing silently#600
sanrai merged 2 commits into
mainfrom
fix/qa-llm-session-id

Conversation

@sanrai

@sanrai sanrai commented Aug 26, 2026

Copy link
Copy Markdown
Collaborator

What broke

On #591 the deterministic layer worked perfectly — build-output-diff published NO_CHANGE and review-score-gate auto-passed the PR with auto-pass: deps-dev + build-output-diff NO_CHANGE (bundle byte-identical).

No agent ever said so. Feature QA Review — the one wired to report exactly that — died at its first LLM call (run 32946357057, job 98107789143):

[llm] transport attempt 1 failed: HTTP 403: {"error":{"type":"missing_required_header","message":"Missing required header: x-session-id"}}
[llm] transport attempt 2 failed: HTTP 403: ... (same)
[llm] transport attempt 3 failed: HTTP 403: ... (same)
feature-review error: Error: llm failed after retries
    at llm (feature-review.mjs:156:9)
    at async feature-review.mjs:306:18

The proxy started requiring x-session-id. Only qa-runner-v2.mjs ever sent it — which is why Agent QA Review kept working (it shells out to that runner) while every direct caller began 403ing.

Then .catch(e => { console.error(...); process.exit(0) }) at the bottom of feature-review.mjs turned the crash into a green job with no comment and no notification. Nothing anywhere said the run had failed.

What this changes

1. Send the header. x-session-id (one randomUUID() per process, same shape as qa-runner-v2.mjs) added to all eight direct proxy callers:

feature-review.mjs · feature-backtest-worker.mjs · feature-backtest-batch.mjs · action-judge.mjs · ai-judge-test.mjs · mobile-probe.mjs · plan-probe.mjs · aggregate-report.mjs

2. Stop gating a deterministic fact behind a model. bundleDiffVerdict() is a commit-status read over the REST API — it needs no LLM at all. But it only runs after the triage LLM call, so a dead proxy took the bundle-diff report down with it. The triage call is now wrapped: on transport failure the agent falls through to postNonInjectable() and still reports SAFE (no bundle change) or NEEDS REVIEW (bundle changed).

That alone would have given #591 the comment it should have had.

3. Make silent outages visible. feature-review.mjs now writes a FEATURE_REVIEW_FAILED marker on an unhandled error, and qa-feature-review.yml logs it to the shared CI: AI / Agent review tool failures issue — the same monitor qa-agent-review.yml has had all along. The PR check still stays green; the failure just stops being invisible.

Notes

  • No product code touched — .github/qa/ and one workflow only.
  • .github/qa/ is outside the npm run lint path (react/src/js/components/Consonant), so no lint impact.
  • No new module, so qa-feature-review.yml's explicit reviewer copy-list is unchanged.
  • Verified with node --check across .github/qa/*.mjs and a YAML parse of the workflow.

Testing

The real check is the next run of this workflow on any PR: Feature QA Review should post a comment again instead of finishing green and silent. Worth a workflow_dispatch against #591 once this lands.

… QA failing silently

The proxy now rejects requests without x-session-id (HTTP 403 missing_required_header). Only qa-runner-v2.mjs sent it, so Agent QA Review kept working while every direct caller started 403ing.

On PR #591 that killed feature-review.mjs at its first LLM call: three 403s, llm failed after retries, and the top-level catch exited 0. Green job, no comment - even though build-output-diff had already published NO_CHANGE and review-score-gate had auto-passed the PR.

- add x-session-id to all eight direct proxy callers

- feature-review.mjs: fall back to postNonInjectable when the triage model is unavailable, so the deterministic bundle-diff verdict is still reported (it is a REST status read, it never needed a model)

- feature-review.mjs: write FEATURE_REVIEW_FAILED on an unhandled error

- qa-feature-review.yml: log that marker to the shared monitor issue, mirroring qa-agent-review.yml, so a silent outage is visible

Claude-Session: https://claude.ai/code/session_01QimdQaYPSvXAaSDAssrfK1
@github-actions

github-actions Bot commented Aug 26, 2026

Copy link
Copy Markdown

Agent QA review — interactive + visual diff (advisory, non-blocking)

Last updated Aug 26, 2026, 10:25 PM PDT · new commit pushed · commit 8935a06 · 9 files changed.

0 open · 0 resolved · visual diff 0.00% · verdict PASS

Open findings

  • Smoke test passed ✅
What the agent checked

Scope check: PR #600 only touches CI/QA tooling scripts (.github/qa/*.mjs, workflow yml) that add an x-session-id header to internal LLM-proxy calls used by automated review bots. It does not touch any frontend/CaaS rendering code. This matches the captured pixel diff showing 0.00% changed pixels — loading the diff.png confirmed no magenta/highlighted regions anywhere on the page, i.e. no visual regions were altered by this PR.

Manual QA performed on https://business.adobe.com/resources/main.html with the PR build injected:

  1. Initial page load — header, breadcrumb, "Explore all Adobe resources" heading, results count (3844 results), sort dropdown, and card grid all rendered correctly, no layout shifts or broken images.
  2. Left filter panel ("Refine Your Results") — rendered correctly with Search box, Products/Content Type/Industry accordion filters.
  3. Typed "AI" into the filter search box — input accepted text and showed a clear (x) button; clicking the clear button correctly reset the field. (Result count did not visibly re-filter in the time observed, but this is unrelated to the PR — the PR does not touch client-side search/filter logic at all, only backend CI scripts.)
  4. Expanded the "Products" filter accordion — chevron flipped, list of checkboxes (Acrobat, Acrobat Sign, Advertising, Analytics, Brand Visibility, Campaign, Commerce, Creative Cloud, Customer Journey Analytics, Forms, etc.) rendered cleanly, properly aligned, no overlap or truncation.
  5. Cards in the grid (Forrester report, AI-powered document workflows, etc.) displayed correctly with images, badges, titles, descriptions, and CTA buttons ("Read now", "Watch now") intact.
  6. run_axe on the filter panel: 0 violations, 22 passes, 0 incomplete — no accessibility regressions.
  7. get_console_errors: only a benign "Permissions policy violation: unload is not allowed in this document" message, unrelated to this PR and not a functional error.

Conclusion: The PR is purely internal CI/QA-automation plumbing (adding a required header to LLM proxy calls so automated review bots stop 403ing and silently exiting 0). It has zero effect on the shipped CaaS bundle or page rendering, consistent with the 0.00% pixel diff. All interactive elements on the live resource page (filters, search box, cards, accessibility) work as expected with no visible breakage, misalignment, or console errors introduced.

Verdict: PASS — no regressions found; PR content is CI-only and the live page behaves identically to stable.

PR / stable / diff screenshots + console + axe artifacts in the workflow run.

Review history (2 runs)
  • 8935a06 · Aug 26, 2026, 10:25 PM PDT · new commit pushed · passing, no regressions — fix(mwpw-184989): send slicc version to LLM proxy
  • 69f4658 · Aug 26, 2026, 1:54 AM PDT · PR opened · passing, no regressions — fix(mwpw-184989): send x-session-id to the LLM proxy and sto

@github-actions

github-actions Bot commented Aug 26, 2026

Copy link
Copy Markdown

🧪 Feature QA review — injected feature test (advisory, non-blocking)

Last updated Aug 26, 2026, 10:21 PM PT · commit 532b68b · pull_request.

Overall: SKIPPED

Not an injectable feature -- skipped.

This PR only fixes CI/tooling scripts (adding an x-session-id header to LLM proxy calls and error handling), with no config- or card-data-driven UI behavior to exercise.

This PR's change isn't driven by config/collection data the harness can force, and the build-output-diff verdict wasn't available to consult. (The visual/smoke review still applies.)

Screenshot in the workflow run.

Review history (2 runs)
  • Aug 26, 2026, 10:21 PM PT · SKIPPED · 532b68b · pull_request
  • Aug 26, 2026, 1:56 AM PT · SKIPPED · 694b863 · pull_request

@github-actions

github-actions Bot commented Aug 26, 2026

Copy link
Copy Markdown

AI Code Review

Last updated Aug 26, 2026 10:22 PM PDT · new commit pushed · commit 8935a06 · 9 files changed in PR.

1 open · 0 resolved

Open findings

  • 🟠 Triage failure fallback doesn't actually report bundle-diff verdict (open since 69f4658)
    In feature-review.mjs, the catch block around the llm() triage call posts a static message claiming to report 'the deterministic bundle diff only' via postNonInjectable(), but no bundle-diff computation or verdict content is actually fetched or included in that message, so the promised fallback signal is not delivered.
Review history (2 runs)
  • 8935a06 · Aug 26, 2026 10:22 PM PDT · new commit pushed · no changes (1 open) — fix(mwpw-184989): send slicc version to LLM proxy
  • 69f4658 · Aug 26, 2026 1:56 AM PDT · PR opened · +1 new (1 open) — fix(mwpw-184989): send x-session-id to the LLM proxy and stop Feature…

@sonarqube-acom-pr-deco-public

Copy link
Copy Markdown

@sanrai
sanrai merged commit faaa4fd into main Aug 27, 2026
20 of 21 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants