[POC TEST] Canary Report — v0.49.0-rc.1 — 2026-05-12T16:30:00Z

[raissanjay]

Canary Report — Cycle 2026-05-12T16:30:00Z

Stable baseline: v0.48.3 (loaded from /stable/main.min.js)
Canary version: v0.49.0-rc.1 (loaded from /v0.49.0-rc.1/main.min.js)
Cohort split: 1% canary, 99% stable (canary diverts 1% of stable's normal traffic for the duration)
Sample window: 2026-05-12T16:30:00Z → 2026-05-12T17:30:00Z — same window for both cohorts
Comparison method: rate metrics aggregated across pages; per-page values reported per page only.

---

Volume

Cohort	Sessions	Users (distinct stickyId)	Beacon sample rate
stable	990	887	1% kept
canary	1000	952	100% kept

---

Summary — rate metrics

Thresholds: errors/session ≤ +20%; render rate ≥ 99% of stable; no per-assertion drop > 2 pp; fetch_fail ≤ +0.2 pp absolute.

Metric	Stable	Canary	Δ	Status
Errors / session (mean count)	0.018	0.025	+37.5%	⚠️ FAIL
Sessions with ≥1 error	1.7%	2.3%	+0.6 pp	✓
cards_rendered fired	99.4%	99.4%	+0.0 pp	✓
target_missing fired	0.6%	0.6%	-0.0 pp	✓
fetch_fail fired	0.1%	0.4%	+0.3 pp	⚠️ MARGINAL

Per-page size/duration metrics (cardCount, TTFC, fetch duration) appear in the per-page table below — they are not meaningful in aggregate because they vary by page type.

---

Assertions — per-session pass rates

Check	Stable	Canary	Δ	Status
Container present	100.0%	100.0%	+0.0 pp	✓
Container has children	99.4%	99.4%	+0.0 pp	✓
All cards visible	99.7%	99.5%	-0.2 pp	✓
Filter panel match	99.8%	95.2%	-4.6 pp	⚠️ FAIL
Paginator match	99.9%	99.8%	-0.1 pp	✓
Sort match	99.9%	99.9%	+0.0 pp	✓
Search match	99.9%	99.9%	+0.0 pp	✓
Loader cleared	99.6%	99.4%	-0.2 pp	✓
No error UI shown	99.9%	99.7%	-0.2 pp	✓

---

Per-page breakdown (top pages by canary volume)

Page	Sessions (s/c)	Errors (s/c)	Filter match (s/c)	Median cardCount (s/c)	Median TTFC ms (s/c)
/products/catalog.html	172 / 178	0 / 0	100.0% / 100.0%	24 / 24	980 / 1010
/creativecloud.html	128 / 135	4 / 8	99.2% / 87.4%	18 / 18	1100 / 1240
/summit/sessions.html	122 / 118	2 / 0	100.0% / 100.0%	36 / 36	1320 / 1350
/products/photoshop.html	94 / 96	1 / 1	99.0% / 95.8%	12 / 12	1050 / 1090

---

Top issues — canary cohort, grouped by signature

Issue #1 — TypeError on main.js:1234

Cannot read property 'cards' of undefined

Field	Value
Source	main.js:1234:42
Error type	TypeError
Stack frame	processCards → fetchCards → setCards
Canary occurrences	8
Stable occurrences	0
Affected canary users	7
Affected pages	/creativecloud.html (8)

Issue #2 — filterPanelMatch: expected but not rendered

Field	Value
Assertion	filterPanelMatch: false
Configured (from CaaS config)	enabled=true, type=left
Observed (DOM check)	.consonant-LeftFilters not present
Canary occurrences	4
Stable occurrences	0
Affected canary users	4
Affected pages	/creativecloud.html (3), /products/photoshop.html (1)

Issue #3 — fetch_fail on https://www.adobe.com/api/cards/cc-collection

Field	Value
URL	https://www.adobe.com/api/cards/cc-collection
HTTP status	500
Median responseTimeMs	340
Canary occurrences	2
Stable occurrences	1
Affected canary users	2

---

Verdict

⚠️ REVIEW BEFORE PROMOTION — 3 threshold breaches detected.

Threshold breaches

• Errors / session: +37.5% (threshold: ≤ +20%) — FAIL
• Filter panel match: -4.6 pp (threshold: ≥ -2 pp) — FAIL
• fetch_fail rate: +0.3 pp (threshold: ≤ +0.2 pp absolute) — MARGINAL

Page concentration of failures

• /creativecloud.html: 11 failures
• /products/photoshop.html: 1 failures

Sample size

• 1000 canary sessions / 952 distinct users — adequate for rate-metric comparisons at the thresholds above.

Review the data above and form a verdict.

cc @sanrai

[cmiqueo]

💡 [AI generated] Recommended Functionality

To improve the utility of these reports, consider adding the following features:

• Automated Rollback Triggers: Since the report already calculates a "Verdict," you could implement a "Circuit Breaker" that automatically stops the canary traffic split if any "FAIL" status is detected, preventing further impact on the 1% cohort.
• Visual Diffing: For the "Filter panel match" failure, adding a functionality that captures a screenshot of the stable vs. canary render when an assertion fails would help debug CSS or DOM-presence issues much faster.
• Latency Percentiles ($P_{95}$ / $P_{99}$): Currently, you only report Median TTFC. Adding $P_{95}$ latency would help you catch performance "jitter" that the median often hides, especially for users on slower connections.
• User Segment Breakdown: It would be helpful to see if errors are browser-specific (e.g., only occurring on Safari) or locale-specific, which would help narrow down the TypeError.