fix(replay-vision): pre-launch hardening sweep - #75874
Merged
Merged
Conversation
Drops the removed emits_embeddings option, sizes scanners in credits rather than observation counts, and gates the vision-actions MCP tools on the replay-vision-actions flag their endpoints actually require.
…evaluate paths Retry now claims its enqueue slot before deleting the failed row, so a capped retry is a pure no-op instead of cascading away the team's rating, and a restore that collides with the replacement run reports 409 rather than 500. Evaluate takes the same row lock apply and dismiss use, and freezes started_at and the model into the workflow inputs so a concurrent re-test can't re-key receipts or re-price the in-flight reservation. Also declares the error-response schemas on retry/observe/suggest_tags/run, standardizes those bodies on detail, and fixes the admin changelist N+1.
A recovery persists synthesized_markdown too, so a retried attempt after it committed reported FIRED and delivered the notification the recovered branch suppresses. The recovered output marker now decides which status the guard returns.
Declares heartbeat timeouts on the evaluation workflow's provider activities and on the reaper, which now describes concurrently and beats between phases instead of burning its tick on a slow Temporal API. The same pass settles prompt-suggestion evaluations stuck in running, which the workflow's swallowed finalize leaves behind. Signal emission gets a 2-minute budget and retries, made safe by a deterministic per-finding idempotency key, so a slow facade no longer records signals_count=0. Event fetching caps at 50k rows and tells the prompt when it truncated.
…stions Tag ops are validated before use and slug-deduped on add, so a malformed op no longer loses an otherwise usable suggestion and an add can't produce a recommendation that always fails to apply. Feedback text is capped in both briefing paths, dismissed prompts can't break out of their fence, evaluation comparison counts freeform tags, the intent/outcome fallback is type-guarded, and the scorer states its effective scale when none is stored.
The dock's scanner rows disable while an observation is starting and the listener rejects a duplicate that beats the re-render, its list load takes a breakpoint so a slow poll response can't resurrect a settled card, and rating mutations drop a response whose epoch has moved on. Loaders that bailed on a missing team now dispatch failure instead of leaving the spinner up, and the run tab's per-session status fetch asks for a limit that leaves room for retried observations.
Collapses the duplicated tag-op ladders in the classifier proposer onto one
transitions generator, extracts the reaper's repeated describe fan-out and runs
its two passes concurrently, and moves the scorer's default scale next to the
ScoreScale contract it has to satisfy.
Retry's quota and capacity pre-checks move off the held row lock, the label
write locks with only("pk") instead of hydrating the row's JSONB, and the
comment blocks come down to one or two lines each.
# Conflicts: # products/replay_vision/backend/api/observations.py # products/replay_vision/backend/temporal/activities/reap_orphaned_observations.py # products/replay_vision/backend/tests/test_reconciler.py
|
😎 Merged manually by @TueHaulund - details. |
Contributor
🤖 CI report
|
| Root | Eager (shipped) | Δ vs base | Budget |
|---|---|---|---|
entry (logged-out pages, app bootstrap)src/index.tsx |
1.25 MiB · 22 files | 🟢 -244 B (-0.0%) | ███░░░░░░░ 27.7% of 4.51 MiB |
authenticated shell (every logged-in page)src/scenes/AuthenticatedShell.tsx |
8.13 MiB · 3,023 files | 🔺 +21 B (+0.0%) | ████████░░ 83.7% of 9.71 MiB |
🟢 node_modules/monaco-editor/ stays out of src/index.tsx
🟢 src/lib/components/ActivityLog/describers stays out of src/index.tsx
🟢 [object Object] stays out of src/index.tsx
🟢 [object Object] stays out of src/index.tsx
🟢 node_modules/monaco-editor/ stays out of src/scenes/AuthenticatedShell.tsx
🟢 src/lib/components/ActivityLog/describers stays out of src/scenes/AuthenticatedShell.tsx
🟢 [object Object] stays out of src/scenes/AuthenticatedShell.tsx
🟢 [object Object] stays out of src/scenes/AuthenticatedShell.tsx
Largest files eagerly shipped from src/index.tsx
| Size | File |
|---|---|
| 126.8 KiB | ../node_modules/.pnpm/react-dom@18.3.1_react@18.3.1/node_modules/react-dom/cjs/react-dom.production.min.js |
| 24.6 KiB | ../node_modules/.pnpm/buffer@6.0.3/node_modules/buffer/index.js |
| 6.3 KiB | ../node_modules/.pnpm/react@18.3.1/node_modules/react/cjs/react.production.min.js |
| 4.5 KiB | ../node_modules/.pnpm/@jspm+core@2.1.0/node_modules/@jspm/core/nodelibs/browser/process.js |
| 3.9 KiB | ../node_modules/.pnpm/scheduler@0.23.2/node_modules/scheduler/cjs/scheduler.production.min.js |
| 1.4 KiB | ../node_modules/.pnpm/base64-js@1.5.1/node_modules/base64-js/index.js |
| 1.3 KiB | src/RootErrorBoundary.tsx |
| 912 B | ../node_modules/.pnpm/ieee754@1.2.1/node_modules/ieee754/index.js |
| 789 B | src/scenes/ChunkLoadErrorBoundary.tsx |
| 762 B | src/index.tsx |
Largest files eagerly shipped from src/scenes/AuthenticatedShell.tsx
| Size | File |
|---|---|
| 285.6 KiB | ../node_modules/.pnpm/posthog-js@1.409.4/node_modules/posthog-js/dist/rrweb.js |
| 267.7 KiB | ../node_modules/.pnpm/@posthog+icons@0.38.0_react-dom@18.3.1_react@18.3.1__react@18.3.1/node_modules/@posthog/icons/dist/posthog-icons.es.js |
| 235.5 KiB | src/taxonomy/core-filter-definitions-by-group.json |
| 231.4 KiB | ../node_modules/.pnpm/posthog-js@1.409.4/node_modules/posthog-js/dist/module.js |
| 154.3 KiB | ../node_modules/.pnpm/re2js@0.4.1/node_modules/re2js/build/index.esm.js |
| 126.8 KiB | ../node_modules/.pnpm/react-dom@18.3.1_react@18.3.1/node_modules/react-dom/cjs/react-dom.production.min.js |
| 105.2 KiB | src/lib/api.ts |
| 94.7 KiB | ../packages/quill/packages/quill/dist/index.js |
| 93.3 KiB | ../node_modules/.pnpm/prosemirror-view@1.40.1/node_modules/prosemirror-view/dist/index.js |
| 90.6 KiB | ../node_modules/.pnpm/@tiptap+core@3.20.6_@tiptap+pm@3.20.6/node_modules/@tiptap/core/dist/index.js |
Posted automatically by check-eager-graph · sizes are eager output bytes (shipped, post-tree-shake) from the esbuild metafile · part of #32479
✅ Toolbar bundle — eager 2.19 MiB within budget
What the toolbar ships to customer pages, measured from the esbuild output (minified, post-tree-shake). The eager set is the entry plus everything statically imported from it — fetched before any feature runs; deferred chunks load lazily. The eager guardrail is 5.72 MiB. Each output file must also stay below 10 MB, where CloudFront stops compressing it. The module boundary is enforced separately by check-toolbar-graph.
| Metric | Size | Δ vs base | Budget |
|---|---|---|---|
| Eager (shipped) entry + static imports |
2.19 MiB · 17 files | 🟢 -386 B (-0.0%) | ████░░░░░░ 38.3% of 5.72 MiB |
| Deferred (lazy) | 2.08 MiB · 33 files | no change | n/a — loads on demand |
Loader dist/toolbar.js |
1.1 KiB | no change | █░░░░░░░░░ 5.8% of 19.5 KiB |
Largest eagerly-shipped chunks
| Size | File |
|---|---|
| 717.7 KiB | dist/toolbar/toolbar-app-RHGTKKCY.css |
| 551.2 KiB | dist/toolbar/chunk-chunk-LMAMJSO4.js |
| 484.6 KiB | dist/toolbar/chunk-chunk-I33HJPDC.js |
| 133.6 KiB | dist/toolbar/chunk-chunk-KNWVTMEH.js |
| 131.8 KiB | dist/toolbar/chunk-chunk-T5KY5WYR.js |
| 71.0 KiB | dist/toolbar/toolbar-app-SA5LAQYV.js |
| 69.0 KiB | dist/toolbar/chunk-chunk-27JL52RE.js |
| 35.6 KiB | dist/toolbar/chunk-chunk-AOSTOGBQ.js |
| 20.9 KiB | dist/toolbar/chunk-chunk-PPPVIVMR.js |
| 12.2 KiB | dist/toolbar/chunk-chunk-PIK3PADE.js |
Posted automatically by check-toolbar-size · sizes are toolbar output bytes (shipped, post-tree-shake) from the esbuild metafile
✅ Dist folder size — 🔺 +6.8 KiB (+0.0%)
Total size of the built frontend/dist folder (all assets), compared against the base branch.
Total: 1373.78 MiB · 🔺 +6.8 KiB (+0.0%)
ℹ️ MCP UI apps size — 32 app(s), 17069.0 KB JS
Built size of each MCP UI app (main.js + styles.css).
| App | JS | CSS |
|---|---|---|
| debug | 599.6 KB | 187.7 KB |
| action | 457.8 KB | 187.7 KB |
| action-list | 564.4 KB | 187.7 KB |
| cohort | 456.8 KB | 187.7 KB |
| cohort-list | 563.4 KB | 187.7 KB |
| email-template | 456.6 KB | 187.7 KB |
| error-details | 472.4 KB | 187.7 KB |
| error-issue | 457.5 KB | 187.7 KB |
| error-issue-list | 564.3 KB | 187.7 KB |
| experiment | 561.5 KB | 187.7 KB |
| experiment-list | 565.2 KB | 187.7 KB |
| experiment-results | 563.2 KB | 187.7 KB |
| feature-flag | 567.2 KB | 187.7 KB |
| feature-flag-list | 570.9 KB | 187.7 KB |
| feature-flag-testing | 461.0 KB | 187.7 KB |
| insight-actors | 562.2 KB | 187.7 KB |
| invite-email-preview | 456.0 KB | 187.7 KB |
| llm-costs | 559.5 KB | 187.7 KB |
| session-recording | 458.6 KB | 187.7 KB |
| session-summary | 463.9 KB | 187.7 KB |
| survey | 458.4 KB | 187.7 KB |
| survey-global-stats | 562.2 KB | 187.7 KB |
| survey-list | 565.1 KB | 187.7 KB |
| survey-stats | 562.2 KB | 187.7 KB |
| trace-span | 457.2 KB | 187.7 KB |
| trace-span-list | 564.3 KB | 187.7 KB |
| workflow | 457.1 KB | 187.7 KB |
| workflow-list | 563.7 KB | 187.7 KB |
| loops-review | 461.2 KB | 187.7 KB |
| query-results | 746.8 KB | 187.7 KB |
| render-ui | 827.4 KB | 187.7 KB |
| visual-review-snapshots | 461.6 KB | 187.7 KB |
⚠️ Backend coverage — 96.0% of changed backend lines covered — 12 uncovered
🧪 Backend test coverage
Patch coverage — changed backend lines (products + core): ███████████████████░ 96.0% (289 / 301)
| File | Patch | Uncovered changed lines |
|---|---|---|
products/replay_vision/backend/observation_formatting.py |
0.0% | 41–42 |
products/replay_vision/backend/api/observations.py |
88.9% | 907, 924–925, 928 |
products/replay_vision/backend/temporal/activities/reap_orphaned_observations.py |
89.8% | 63–64, 99, 102, 157 |
products/replay_vision/backend/prompt_suggestions.py |
90.9% | 129 |
🤖 Agents: add a test covering the lines above, or note why under "How did you test this code?". Machine-readable gap list: the patch-coverage artifact on this run (gh run download 30637472224 -n patch-coverage), or the coverage-data block at the end of this comment.
Per-product line coverage (touched products)
| Product | Coverage | Lines |
|---|---|---|
platform_features |
██░░░░░░░░░░░░░░░░░░ 12.1% |
7 / 58 |
batch_exports |
████████░░░░░░░░░░░░ 39.4% |
8,789 / 22,299 |
demo |
███████████░░░░░░░░░ 56.3% |
1,497 / 2,661 |
warehouse_sources_queue |
████████████░░░░░░░░ 59.2% |
148 / 250 |
data_tools |
██████████████░░░░░░ 70.0% |
63 / 90 |
tasks |
██████████████░░░░░░ 70.1% |
32,888 / 46,903 |
ai_gateway |
███████████████░░░░░ 75.0% |
9 / 12 |
signals |
████████████████░░░░ 81.3% |
24,976 / 30,718 |
cdp |
████████████████░░░░ 82.1% |
3,285 / 3,999 |
data_modeling |
█████████████████░░░ 85.7% |
7,716 / 9,008 |
notebooks |
█████████████████░░░ 86.0% |
7,794 / 9,060 |
wizard |
█████████████████░░░ 86.4% |
1,060 / 1,227 |
actions |
█████████████████░░░ 86.6% |
717 / 828 |
cohorts |
██████████████████░░ 87.8% |
6,181 / 7,040 |
product_tours |
██████████████████░░ 87.9% |
1,303 / 1,482 |
exports |
██████████████████░░ 88.2% |
7,046 / 7,986 |
data_warehouse |
██████████████████░░ 88.4% |
12,192 / 13,798 |
business_knowledge |
██████████████████░░ 89.0% |
4,391 / 4,936 |
engineering_analytics |
██████████████████░░ 89.3% |
6,529 / 7,309 |
conversations |
██████████████████░░ 89.3% |
17,600 / 19,701 |
dashboards |
██████████████████░░ 89.5% |
5,989 / 6,693 |
visual_review |
██████████████████░░ 89.5% |
5,870 / 6,558 |
alerts |
██████████████████░░ 90.3% |
4,482 / 4,966 |
links |
██████████████████░░ 90.6% |
183 / 202 |
streamlit_apps |
██████████████████░░ 90.7% |
2,630 / 2,901 |
mcp_analytics |
██████████████████░░ 90.9% |
3,191 / 3,511 |
error_tracking |
██████████████████░░ 91.0% |
10,928 / 12,010 |
slack_app |
██████████████████░░ 91.1% |
9,664 / 10,610 |
marketing_analytics |
██████████████████░░ 91.2% |
12,092 / 13,265 |
stamphog |
██████████████████░░ 91.3% |
4,505 / 4,936 |
mcp_store |
██████████████████░░ 91.9% |
4,273 / 4,650 |
product_analytics |
███████████████████░ 92.5% |
5,849 / 6,321 |
managed_migrations |
███████████████████░ 92.6% |
1,556 / 1,681 |
early_access_features |
███████████████████░ 92.6% |
1,287 / 1,390 |
notifications |
███████████████████░ 92.6% |
1,017 / 1,098 |
ai_observability |
███████████████████░ 92.8% |
15,618 / 16,821 |
surveys |
███████████████████░ 93.1% |
5,771 / 6,197 |
posthog_ai |
███████████████████░ 93.2% |
1,326 / 1,422 |
web_analytics |
███████████████████░ 93.3% |
14,911 / 15,976 |
approvals |
███████████████████░ 93.3% |
3,437 / 3,682 |
reminders |
███████████████████░ 93.4% |
468 / 501 |
legal_documents |
███████████████████░ 93.8% |
1,628 / 1,736 |
workflows |
███████████████████░ 94.1% |
7,241 / 7,698 |
endpoints |
███████████████████░ 94.2% |
8,655 / 9,192 |
tracing |
███████████████████░ 94.5% |
2,671 / 2,827 |
review_hog |
███████████████████░ 94.6% |
8,101 / 8,563 |
skills |
███████████████████░ 94.6% |
3,158 / 3,337 |
messaging |
███████████████████░ 94.7% |
2,885 / 3,048 |
logs |
███████████████████░ 95.4% |
10,466 / 10,966 |
experiments |
███████████████████░ 95.5% |
26,047 / 27,287 |
growth |
███████████████████░ 96.1% |
3,245 / 3,376 |
annotations |
███████████████████░ 96.2% |
732 / 761 |
revenue_analytics |
███████████████████░ 96.3% |
1,887 / 1,960 |
feature_flags |
███████████████████░ 96.4% |
17,488 / 18,144 |
replay_vision |
███████████████████░ 96.4% |
16,286 / 16,891 |
user_interviews |
███████████████████░ 96.5% |
2,638 / 2,734 |
access_control |
███████████████████░ 96.9% |
870 / 898 |
customer_analytics |
███████████████████░ 97.1% |
10,444 / 10,758 |
warehouse_sources |
███████████████████░ 97.3% |
359,732 / 369,739 |
data_catalog |
████████████████████ 97.7% |
2,588 / 2,648 |
analytics_platform |
████████████████████ 98.0% |
2,153 / 2,197 |
metrics |
████████████████████ 98.2% |
2,491 / 2,536 |
pulse |
████████████████████ 98.4% |
2,017 / 2,049 |
live_debugger |
████████████████████ 99.2% |
613 / 618 |
field_notes |
████████████████████ 99.4% |
158 / 159 |
Report-only. Patch coverage = changed backend lines covered vs origin/master. Sorted lowest first.
Known gaps: lines covered only by Temporal tests show as uncovered; core line numbers may drift if master changed the same file.
TueHaulund
requested review from
a team,
arnohillen,
fasyy612 and
ksvat
and removed request for
a team
July 31, 2026 08:42
TueHaulund
marked this pull request as ready for review
July 31, 2026 08:42
Contributor
🦔 Hogbox preview · ✅ ready▶ Open the preview
commit |
Contributor
Prompt To Fix All With AI### Issue 1
products/replay_vision/backend/api/observations.py:916
**Retry compensation drops labels**
When workflow startup returns `FAILED` or `ALREADY_RUNNING`, deleting the observation cascades to its shared label, but the compensation path recreates only the observation row. The endpoint then reports that the failed observation was kept while the team's rating and feedback have been permanently lost.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.Reviews (1): Last reviewed commit: "Merge branch 'master' into tue/rv-prelau..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
…'t start The delete that frees the UNIQUE(scanner, session_id) slot cascades the shared label away. Restoring only the row left the team's rating and feedback gone while the 503 said the observation was kept. The label is now captured before the delete and re-inserted with the row, original timestamps included.
… the new mcp flag The retry lock looked the row up by bare pk, which trips semgrep's idor-lookup-without-team (the create_task lock is exempt only because it passes pk=observation.pk). Not reachable across teams since get_object() already scoped it, but the query should say so. Scopes the label lock the same way. replay-vision-actions is a new gating flag, so the MCP flag-inventory test counts 27 rather than 26.
arnohillen
approved these changes
Jul 31, 2026
arnohillen
left a comment
arnohillen
left a comment
Contributor
There was a problem hiding this comment.
Went through all six fix categories (retry/evaluate races, alert recovery, temporal heartbeats and reaper, LLM output handling, frontend guards, MCP flag gating) plus git-history and past-PR-feedback sweeps: no issues found, and the error-body standardization to {detail} is consistent across endpoints. One logistics note: the branch currently has merge conflicts with master, so it needs a rebase before it can merge.
# Conflicts: # products/replay_vision/mcp/tools.yaml # services/mcp/schema/generated-tool-definitions.json # services/mcp/schema/tool-definitions-all.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Replay Vision ships Monday. A pre-launch review turned up a set of races, dead-ends and stale docs. Individually small, all reachable from normal use. No new features here.
Changes
One commit per group.
Retry and evaluate races. Retry deleted the failed observation before knowing the replacement run could start, so two concurrent retries both passed the status check, and a quota-capped retry deleted the row (cascading away the team's rating) for a request that changed nothing. It now claims the enqueue slot before deleting, under a row lock, and a restore that collides with the replacement run reports 409 instead of 500. Prompt-suggestion
evaluatetakes the same lockapplyanddismissuse, and freezesstarted_atand the model into the workflow inputs so a concurrent re-test can't re-key usage receipts or re-price the reservation.Alert recovery.
_persist_recoveredwritessynthesized_markdowntoo, so a retried attempt after it committed re-reported FIRED and delivered the notification recovering exists to suppress. Therecoveredmarker now decides what the guard returns.Temporal. Heartbeat timeouts on the evaluation provider activities and the reaper. The reaper now also settles prompt-suggestion evaluations stuck in
running, which the workflow's swallowed finalize leaves behind forever. Signal emission gets 2 minutes and retries, safe via a deterministic per-finding idempotency key, so a slow facade stops recordingsignals_count=0. Event fetching caps at 50k rows and tells the prompt it truncated.LLM output. Tag ops validated before use and slug-deduped on add. Feedback capped in both briefing paths, dismissed prompts can't break their fence, evaluation comparison counts freeform tags.
Frontend. Dock rows disable while an observation starts and the listener rejects a duplicate that beats the re-render. List load takes a
breakpoint(). Rating mutations drop a stale-epoch response. Loaders that bailed on a missing team dispatch failure instead of spinning forever.Docs and MCP. The scanner skill documented a removed
emits_embeddingsoption and sized scanners in observations when the quota is credits. The fourvision-actions-*tools were gated onreplay-visionwhile their endpoints requirereplay-vision-actions, so they appeared and then 404'd.Note
Vision Actions owners: please look at
backend/temporal/vision_actions/alerts.pyand thetools.yamlgating. Both kept minimal.Behavior changes:
vision-actions-*MCP tools now needreplay-vision-actions. Intended.observeandsuggest_tags503 bodies go from{"error": ...}to{"detail": ...}. Nothing read.error.Tags: churn, churnbecomesTags: churnandpayment issuesbecomespayment_issues. Right for comparison, but it is user-visible text, so say the word if you want display split from comparison.How did you test this code?
No manual testing. All automated, run locally by the agent: full
products/replay_vision/backend/tests/suite (1219 passed, re-run after cleanup and after merging master), Jestreplay_vision(304 passed), repo-wide mypy clean,hogli ci:preflight --strict0 failures,hogli build:openapiregenerated and committed.New tests and the regression each catches:
started_atit started with, not a re-read of a row a concurrent re-test restamps.tag_ops(bare strings, lists, non-string tags, slug collisions, non-listtag_ops) leave the vocabulary alone and keep the rewritten prompt.Automatic notifications
Docs update
Scanner-creation skill updated in this PR.
🤖 Agent context
Autonomy: Human-driven (agent-assisted)
I gave Claude (Opus 5, Claude Code) a written plan from a pre-launch review and asked it to implement all of it. Skills:
/improving-drf-endpoints,/implementing-mcp-tools,/writing-tests,/writing-kea-logics,/simplify.Two plan items it couldn't follow literally. The dock guard was specced as an early return on
values.observing, but the reducer flips that true before the listener runs, so it rejects every call; it uses a cache flag, with a test that fails without it. The plan also wanted a 409 for retrying a non-FAILED observation while insisting existing response shapes stay; that path is a 400 with a test pinning it, so it stayed 400. Worth a call either way./simplifythen ran four review passes and applied the results (net minus 17 lines): the classifier's duplicated tag-op ladder onto one generator, the reaper's repeated describe fan-out extracted, the scorer's default scale moved next toScoreScale, retry's quota and capacity pre-checks moved off the held row lock. Merging master afterwards was good timing: master had just landed a sharedclassify_stale_rowsthat is exactly what the reaper wanted, so the local helper is gone in favour of it.Left out on purpose:
label.feedbackand LLM reasoning still reach the briefing unfenced; the fence fix here only covers dismissed prompts. Wrapping the briefing inas_untrusted_datachanges every scanner's prompt, so it wants its own PR.trigger.py. Both redesign a freshly-tested path.reap_stuck_vision_action_runssets a cleaner precedent to follow.replay-vision-team-in-flight-capspatch branch and delete the legacycount_in_flight_appliesactivity.