feat(signals): auto-pause scouts that keep failing, and name the wall

[posthog]

Problem

A Signals scout lane, meaning one (team, skill) pair, that can never succeed stays due on every coordinator tick. Each dispatch is well formed, takes a sandbox lease for the whole runtime cap, produces nothing, and books a failed run. Nothing in the harness reconciles "this has never once worked", so the lane re-dispatches forever. The only trace is a stream of individually unremarkable failure events, which is why a project's Signals inbox can sit empty for days while its scouts burn a lease per interval and nobody notices.

Underneath that, every exhausted per-turn poll budget raised the same error string. That makes the fleet's timeout rate read as one cause when it is really three that want opposite fixes: an agent that never emitted a single turn-relevant line (it never got going, so a bigger budget changes nothing), one that worked and then went quiet, and one still streaming when the budget expired (the budget really is the constraint). With one string covering all three, "is the runtime cap too tight?" is not answerable from the failure data.

Changes

Note

This PR was reworked after #75349 (the scout lifecycle status enum) merged. The breaker no longer carries its own pause fields, it is the first consumer of the status primitive.

A failure-streak circuit breaker, expressed through the status enum. One new field, consecutive_failure_count. The runner bumps it on a failed run and zeroes it on a successful one. At FAILURE_STREAK_PAUSE_THRESHOLD the runner pauses the lane through transition_status_by_system (status=paused_by_system, pause_reason=repeated_failures), which also syncs enabled=false like any pause. The helper's reason-scoped ownership rule means the breaker can only lift its own pauses, never a human's paused_by_user and never another writer's reason.

The half-open probe is a reason-scoped coordinator rule. A paused lane drops out of the normal enabled=true dispatch query, so _collect_probe_runs fetches lanes paused with repeated_failures and dispatches one probe per AUTO_PAUSE_PROBE_INTERVAL_S (cooldown anchored on last_run_at, so a failed probe restarts it with no extra bookkeeping). A probe that succeeds resumes the lane through the same helper, and every resume, system or human, starts with a zeroed streak so one fresh failure can't instantly re-trip. Cancelled runs are deliberately not counted, otherwise a few deploys would pause healthy scouts.

The deferred status coherence constraints ride along. #75349 deliberately deferred its DB backstop for the enabled/status dual-write until a full deploy cycle had rolled. That cycle has passed, so this PR carries it: migration 0080 re-runs the enabled-wins reconciliation over rows old pods drifted during the rollout window, then 0081/0082 add and validate two CHECK constraints (enabled matches status, pause_reason set exactly for the pausing statuses) via AddConstraintNotValid + ValidateConstraint. Landing them before this PR's system writer goes live turns any future incoherent write into a loud failure instead of a silent one.

The pause is visible where scouts are managed. consecutive_failure_count is read-only on the config serializers, and the fleet list shows a Paused badge (keyed on status + pause_reason from the primitive's fields) whose tooltip explains the retry behavior and the one-click resume. The trip emits signals_scout_config_auto_paused once per wedge, so a count of that event is a count of wedged lanes; the pause itself also logs through status like any other lifecycle change.

A typed TurnPollTimeout that names the wall (unchanged from the original PR). It classifies the failure as no_turn_output, stalled_after_output, or active_at_budget and carries the turn-log line counts, riding in both the message and structured properties on signals_scout_run_finished.

Note

No timeouts, budgets, or salvage windows changed. The instrumentation exists so the resulting populations can be judged separately.

How did you test this code?

Automated only (agent-run):

• test_scout_status.py (26 passed): two new tests catch a resume that keeps a stale streak, which would let a single fresh failure instantly re-trip the breaker after a resume or human re-enable. The rest are the primitive's existing ownership-rule suite, unchanged.
• test_scout_harness.py (66 passed): the end-to-end breaker test reworked to the enum: streak accumulates, trip pauses with reason and fires the event once, a failed probe does not re-alert, a successful probe resumes clean.
• test_scout_coordinator.py (398 passed): new _collect_probe_runs cases catch wrong reason scoping (a human pause or another writer's pause being probed) and wrong cooldown arithmetic, plus one planning-activity case that catches the probe path being dropped from _collect_planned_runs, without which a fixed lane stays paused forever.
• test_scout_harness_api.py + test_multi_turn_session.py: one new endpoint case locks in that an unrelated config edit resets the streak but does not resume the pause (resuming through an edit would sidestep the enabled-scout cap).
• Migrations: makemigrations --check clean, sqlmigrate verified NOT VALID constraint SQL and a real Postgres default on the new column. Repo-wide mypy clean, frontend typescript:check clean, touched Jest suites (12) pass, hogli ci:preflight --fix 0 failures.

Automatic notifications

• Publish to changelog?
• Alert Sales and Marketing teams?

Docs update

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

Reworked with Claude Code, directed by Andy: the original bot-authored breaker (its own auto_paused_at / auto_pause_reason fields, never touching enabled) was rebased onto the status primitive from #75349 per the coordination plan in this comment, and the primitive's deferred constraint follow-up was folded in rather than shipped as a separate migrations-only PR. Decisions worth knowing: an unrelated config edit resets the streak but no longer clears the pause (the old behavior would bypass the enabled-scout cap that both sanctioned resume paths re-check), and the probe cooldown reuses last_run_at instead of a dedicated timestamp so a failed probe re-arms it for free. Skills invoked: /django-migrations, /improving-drf-endpoints, /writing-tests, /writing-user-facing-copy.

https://claude.ai/code/session_01QXwLjesj6xXp7kwPjnFNFk

[trunk-io]

😎 Merged manually by @andrewm4894 - details.

[github-actions]

🤖 CI report

⚠️ Bundle size — 🔺 +9.5 KiB (+0.0%)

Uncompressed size of every built .js bundle, compared against the base branch.

Total: 65.44 MiB · 🔺 +9.5 KiB (+0.0%)

File	Size	Δ vs base
posthog-app/_parent/products/workflows/frontend/WorkflowsScene.js	77.7 KiB	🔺 +7.0 KiB (+9.9%)
posthog-app/src/scenes/instance/QueryPerformance/QueryPerformance.js	34.7 KiB	🔺 +1.6 KiB (+4.9%)

_{Posted automatically by build-bundle-size-report · uncompressed bytes from dist-report}

✅ Eager graph — within budget

How much code each root ships on the eager path — downloaded and parsed before the surface is interactive. Measured from the esbuild output chunks (post-tree-shake, static imports only); lazy import() / React.lazy chunks are not counted.

Root	Eager (shipped)	Δ vs base	Budget
entry (logged-out pages, app bootstrap) src/index.tsx	1.25 MiB · 22 files	🟢 -20 B (-0.0%)	███░░░░░░░ 27.7% of 4.51 MiB
authenticated shell (every logged-in page) src/scenes/AuthenticatedShell.tsx	8.12 MiB · 3,023 files	🔺 +1.0 KiB (+0.0%)	████████░░ 83.6% of 9.71 MiB

🟢 node_modules/monaco-editor/ stays out of src/index.tsx
🟢 src/lib/components/ActivityLog/describers stays out of src/index.tsx
🟢 [object Object] stays out of src/index.tsx
🟢 [object Object] stays out of src/index.tsx
🟢 node_modules/monaco-editor/ stays out of src/scenes/AuthenticatedShell.tsx
🟢 src/lib/components/ActivityLog/describers stays out of src/scenes/AuthenticatedShell.tsx
🟢 [object Object] stays out of src/scenes/AuthenticatedShell.tsx
🟢 [object Object] stays out of src/scenes/AuthenticatedShell.tsx

Largest files eagerly shipped from src/index.tsx

Size	File
126.8 KiB	../node_modules/.pnpm/react-dom@18.3.1_react@18.3.1/node_modules/react-dom/cjs/react-dom.production.min.js
24.6 KiB	../node_modules/.pnpm/buffer@6.0.3/node_modules/buffer/index.js
6.3 KiB	../node_modules/.pnpm/react@18.3.1/node_modules/react/cjs/react.production.min.js
4.5 KiB	../node_modules/.pnpm/@jspm+core@2.1.0/node_modules/@jspm/core/nodelibs/browser/process.js
3.9 KiB	../node_modules/.pnpm/scheduler@0.23.2/node_modules/scheduler/cjs/scheduler.production.min.js
1.4 KiB	../node_modules/.pnpm/base64-js@1.5.1/node_modules/base64-js/index.js
1.3 KiB	src/RootErrorBoundary.tsx
912 B	../node_modules/.pnpm/ieee754@1.2.1/node_modules/ieee754/index.js
789 B	src/scenes/ChunkLoadErrorBoundary.tsx
762 B	src/index.tsx

Largest files eagerly shipped from src/scenes/AuthenticatedShell.tsx

Size	File
285.3 KiB	../node_modules/.pnpm/posthog-js@1.409.2/node_modules/posthog-js/dist/rrweb.js
267.7 KiB	../node_modules/.pnpm/@posthog+icons@0.38.0_react-dom@18.3.1_react@18.3.1__react@18.3.1/node_modules/@posthog/icons/dist/posthog-icons.es.js
235.5 KiB	src/taxonomy/core-filter-definitions-by-group.json
231.4 KiB	../node_modules/.pnpm/posthog-js@1.409.2/node_modules/posthog-js/dist/module.js
154.3 KiB	../node_modules/.pnpm/re2js@0.4.1/node_modules/re2js/build/index.esm.js
126.8 KiB	../node_modules/.pnpm/react-dom@18.3.1_react@18.3.1/node_modules/react-dom/cjs/react-dom.production.min.js
105.2 KiB	src/lib/api.ts
94.7 KiB	../packages/quill/packages/quill/dist/index.js
93.3 KiB	../node_modules/.pnpm/prosemirror-view@1.40.1/node_modules/prosemirror-view/dist/index.js
90.6 KiB	../node_modules/.pnpm/@tiptap+core@3.20.6_@tiptap+pm@3.20.6/node_modules/@tiptap/core/dist/index.js

_{Posted automatically by check-eager-graph · sizes are eager output bytes (shipped, post-tree-shake) from the esbuild metafile · part of #32479}

✅ Toolbar bundle — eager 2.19 MiB within budget

What the toolbar ships to customer pages, measured from the esbuild output (minified, post-tree-shake). The eager set is the entry plus everything statically imported from it — fetched before any feature runs; deferred chunks load lazily. The eager guardrail is 5.72 MiB. Each output file must also stay below 10 MB, where CloudFront stops compressing it. The module boundary is enforced separately by check-toolbar-graph.

Metric	Size	Δ vs base	Budget
Eager (shipped) entry + static imports	2.19 MiB · 17 files	🔺 +13 B (+0.0%)	████░░░░░░ 38.3% of 5.72 MiB
Deferred (lazy)	2.08 MiB · 33 files	no change	n/a — loads on demand
Loader dist/toolbar.js	1.1 KiB	no change	█░░░░░░░░░ 5.8% of 19.5 KiB

Largest eagerly-shipped chunks

Size	File
717.7 KiB	dist/toolbar/toolbar-app-3ZUOLZ25.css
551.1 KiB	dist/toolbar/chunk-chunk-MGCJX44V.js
484.6 KiB	dist/toolbar/chunk-chunk-QCKKTYHY.js
133.6 KiB	dist/toolbar/chunk-chunk-J5MNYTRE.js
131.8 KiB	dist/toolbar/chunk-chunk-T5KY5WYR.js
71.0 KiB	dist/toolbar/toolbar-app-LF3AKWIJ.js
69.0 KiB	dist/toolbar/chunk-chunk-27JL52RE.js
35.6 KiB	dist/toolbar/chunk-chunk-Z7LVFBRX.js
20.9 KiB	dist/toolbar/chunk-chunk-K5Z37JFK.js
12.2 KiB	dist/toolbar/chunk-chunk-PIK3PADE.js

_{Posted automatically by check-toolbar-size · sizes are toolbar output bytes (shipped, post-tree-shake) from the esbuild metafile}

✅ Dist folder size — 🔺 +152.0 KiB (+0.0%)

Total size of the built frontend/dist folder (all assets), compared against the base branch.

Total: 1371.14 MiB · 🔺 +152.0 KiB (+0.0%)

✅ Playwright — all passed

All tests passed.

View test results →

⚠️ Backend coverage — 94.0% of changed backend lines covered — 15 uncovered

🧪 Backend test coverage

Patch coverage — changed backend lines (products + core): ███████████████████░ 94.0% (267 / 282)

File	Patch	Uncovered changed lines
products/signals/backend/scout_harness/runner.py	75.0%	801, 805–806, 828, 831, 840–842, 854, 993–994, 1081
products/signals/backend/models.py	91.7%	1358
products/signals/backend/temporal/agentic/scout_coordinator.py	95.2%	445
products/tasks/backend/logic/services/custom_prompt_internals.py	97.2%	801

🤖 Agents: add a test covering the lines above, or note why under "How did you test this code?". Machine-readable gap list: the patch-coverage artifact on this run (gh run download 30624441306 -n patch-coverage), or the coverage-data block at the end of this comment.

Per-product line coverage (touched products)

Product	Coverage	Lines
platform_features	██░░░░░░░░░░░░░░░░░░ 12.1%	7 / 58
batch_exports	████████░░░░░░░░░░░░ 39.2%	8,646 / 22,060
demo	███████████░░░░░░░░░ 56.3%	1,497 / 2,661
warehouse_sources_queue	████████████░░░░░░░░ 59.2%	148 / 250
data_tools	██████████████░░░░░░ 70.0%	63 / 90
tasks	██████████████░░░░░░ 70.1%	32,838 / 46,826
ai_gateway	███████████████░░░░░ 75.0%	9 / 12
signals	████████████████░░░░ 81.3%	24,958 / 30,693
cdp	████████████████░░░░ 82.1%	3,285 / 3,999
data_modeling	█████████████████░░░ 85.6%	7,688 / 8,979
notebooks	█████████████████░░░ 86.0%	7,794 / 9,060
wizard	█████████████████░░░ 86.4%	1,060 / 1,227
actions	█████████████████░░░ 86.6%	717 / 828
cohorts	██████████████████░░ 87.8%	6,181 / 7,040
product_tours	██████████████████░░ 87.9%	1,303 / 1,482
exports	██████████████████░░ 88.2%	7,046 / 7,986
data_warehouse	██████████████████░░ 88.4%	12,192 / 13,798
business_knowledge	██████████████████░░ 89.0%	4,391 / 4,936
engineering_analytics	██████████████████░░ 89.3%	6,529 / 7,309
conversations	██████████████████░░ 89.3%	17,600 / 19,701
dashboards	██████████████████░░ 89.4%	5,983 / 6,693
visual_review	██████████████████░░ 89.5%	5,870 / 6,558
alerts	██████████████████░░ 90.2%	4,458 / 4,942
links	██████████████████░░ 90.6%	183 / 202
streamlit_apps	██████████████████░░ 90.7%	2,630 / 2,901
mcp_analytics	██████████████████░░ 90.9%	3,191 / 3,511
error_tracking	██████████████████░░ 91.0%	10,926 / 12,008
slack_app	██████████████████░░ 91.1%	9,664 / 10,610
marketing_analytics	██████████████████░░ 91.2%	12,092 / 13,265
stamphog	██████████████████░░ 91.3%	4,505 / 4,936
mcp_store	██████████████████░░ 91.9%	4,257 / 4,634
product_analytics	███████████████████░ 92.5%	5,849 / 6,321
managed_migrations	███████████████████░ 92.6%	1,556 / 1,681
early_access_features	███████████████████░ 92.6%	1,287 / 1,390
notifications	███████████████████░ 92.6%	1,017 / 1,098
ai_observability	███████████████████░ 92.8%	15,618 / 16,821
surveys	███████████████████░ 93.1%	5,771 / 6,197
posthog_ai	███████████████████░ 93.2%	1,326 / 1,422
web_analytics	███████████████████░ 93.3%	14,911 / 15,976
approvals	███████████████████░ 93.3%	3,437 / 3,682
reminders	███████████████████░ 93.4%	468 / 501
legal_documents	███████████████████░ 93.8%	1,628 / 1,736
workflows	███████████████████░ 94.1%	7,241 / 7,698
endpoints	███████████████████░ 94.2%	8,655 / 9,192
tracing	███████████████████░ 94.5%	2,671 / 2,827
review_hog	███████████████████░ 94.6%	8,101 / 8,563
skills	███████████████████░ 94.6%	3,158 / 3,337
messaging	███████████████████░ 94.7%	2,885 / 3,048
experiments	███████████████████░ 95.4%	25,938 / 27,177
logs	███████████████████░ 95.5%	10,440 / 10,937
growth	███████████████████░ 96.1%	3,245 / 3,376
annotations	███████████████████░ 96.2%	732 / 761
revenue_analytics	███████████████████░ 96.3%	1,887 / 1,960
feature_flags	███████████████████░ 96.4%	17,488 / 18,144
replay_vision	███████████████████░ 96.4%	16,081 / 16,679
user_interviews	███████████████████░ 96.5%	2,638 / 2,734
access_control	███████████████████░ 96.9%	870 / 898
customer_analytics	███████████████████░ 97.1%	10,444 / 10,758
warehouse_sources	███████████████████░ 97.3%	359,272 / 369,318
data_catalog	████████████████████ 97.8%	2,589 / 2,648
analytics_platform	████████████████████ 98.0%	2,153 / 2,197
metrics	████████████████████ 98.2%	2,491 / 2,536
pulse	████████████████████ 98.4%	2,017 / 2,049
live_debugger	████████████████████ 99.2%	613 / 618
field_notes	████████████████████ 99.4%	158 / 159

Report-only. Patch coverage = changed backend lines covered vs origin/master. Sorted lowest first.
Known gaps: lines covered only by Temporal tests show as uncovered; core line numbers may drift if master changed the same file.

⚠️ Django migration SQL — 4 new migrations to review

We've detected new migrations on this PR. Review the SQL output for each migration:

products/signals/backend/migrations/0079_signalscoutconfig_consecutive_failure_count.py

BEGIN;
--
-- Add field consecutive_failure_count to signalscoutconfig
--
ALTER TABLE "signals_signalscoutconfig" ADD COLUMN "consecutive_failure_count" integer DEFAULT 0 NOT NULL CHECK ("consecutive_failure_count" >= 0);
COMMIT;

products/signals/backend/migrations/0080_reconcile_scout_status_enabled.py

BEGIN;
--
-- Raw Python operation
--
-- THIS OPERATION CANNOT BE WRITTEN AS SQL
COMMIT;

products/signals/backend/migrations/0081_scout_status_constraints.py

BEGIN;
--
-- Add CHECK constraint scout_config_enabled_matches_status on signalscoutconfig (NOT VALID)
--
ALTER TABLE "signals_signalscoutconfig" ADD CONSTRAINT "scout_config_enabled_matches_status" CHECK ((("enabled" AND "status" IN ('active', 'pending_pause')) OR (NOT "enabled" AND "status" IN ('paused_by_system', 'paused_by_user')))) NOT VALID;
--
-- Add CHECK constraint scout_config_pause_reason_matches_status on signalscoutconfig (NOT VALID)
--
ALTER TABLE "signals_signalscoutconfig" ADD CONSTRAINT "scout_config_pause_reason_matches_status" CHECK ((("pause_reason" IS NOT NULL AND "status" IN ('pending_pause', 'paused_by_system')) OR ("pause_reason" IS NULL AND "status" IN ('active', 'paused_by_user')))) NOT VALID;
COMMIT;

products/signals/backend/migrations/0082_validate_scout_status_constraints.py

BEGIN;
--
-- Validate constraint scout_config_enabled_matches_status on signalscoutconfig
--
SET lock_timeout = 0;
SET statement_timeout = 0;
ALTER TABLE "signals_signalscoutconfig" VALIDATE CONSTRAINT "scout_config_enabled_matches_status";
--
-- Validate constraint scout_config_pause_reason_matches_status on signalscoutconfig
--
SET lock_timeout = 0;
SET statement_timeout = 0;
ALTER TABLE "signals_signalscoutconfig" VALIDATE CONSTRAINT "scout_config_pause_reason_matches_status";
COMMIT;

Last updated: 2026-07-31 10:45 UTC (19786b3)

✅ Django migration risk — migration analysis complete

We've analyzed your migrations for potential risks.

Summary: 2 Safe | 2 Needs Review | 0 Blocked

⚠️ Needs Review

May have performance impact

signals.0080_reconcile_scout_status_enabled
  └─ #1 ⚠️ RunPython: RunPython data migration needs review for performance

signals.0082_validate_scout_status_constraints
  └─ #1 ⚠️ ValidateConstraint
     VALIDATE CONSTRAINT can be slow but non-blocking (allows reads/writes)
     model: signalscoutconfig, constraint: scout_config_enabled_matches_status
  └─ #2 ⚠️ ValidateConstraint
     VALIDATE CONSTRAINT can be slow but non-blocking (allows reads/writes)
     model: signalscoutconfig, constraint: scout_config_pause_reason_matches_status

✅ Safe

Brief or no lock, backwards compatible

signals.0079_signalscoutconfig_consecutive_failure_count
  └─ #1 ✅ AddField
     Adding NOT NULL field with constant default (safe in PG11+)
     model: signalscoutconfig, field: consecutive_failure_count

signals.0081_scout_status_constraints
  └─ #1 ✅ AddConstraintNotValid
     ADD CONSTRAINT ... NOT VALID is safe (validates new rows only, no table scan)
     model: signalscoutconfig, constraint: scout_config_enabled_matches_status
  └─ #2 ✅ AddConstraintNotValid
     ADD CONSTRAINT ... NOT VALID is safe (validates new rows only, no table scan)
     model: signalscoutconfig, constraint: scout_config_pause_reason_matches_status

📚 How to Deploy These Changes Safely

AddConstraintNotValid:

Follow up with ValidateConstraint in a later migration to check existing rows.

RunPython:

Use batching for large data migrations:

• Use .iterator() to avoid loading all rows into memory
• Use .bulk_update() instead of saving individual objects
• Batch size: 1,000-10,000 rows per batch
• Add pauses between batches
• Consider background jobs for very large updates (millions of rows)

See the migration safety guide

ValidateConstraint:

Long-running on large tables but uses SHARE UPDATE EXCLUSIVE lock (allows normal operations).

Last updated: 2026-07-31 10:46 UTC (19786b3)

[andrewm4894]

Heads-up on direction: the scout lifecycle primitive has now landed separately in #75349 (approved), and the plan agreed in the linked design discussion is for this PR to rebase onto it as a consumer — it and #75053 were about to introduce two independent auto_paused_at / auto_pause_reason field sets with opposite semantics on the same rows.

What changes here, and what it buys:

• Drop this PR's auto_paused_at / auto_pause_reason fields and migration (0075/0076 are also now taken by feat(signals): add scout lifecycle status with reason-scoped pauses #75349, so the file needs renumbering regardless). Keep consecutive_failure_count — the streak is genuinely this PR's.
• Trip the breaker via transition_status_by_system(Status.PAUSED_BY_SYSTEM, pause_reason=PauseReason.REPEATED_FAILURES) — the single entry point for system writers. It runs under a row lock and enforces the reason-scoped ownership rule, so the breaker can never overwrite a human's pause or another writer's (the inactivity sweep's) state, and vice versa.
• The half-open probe becomes a reason-scoped coordinator rule: also fetch rows with status=paused_by_system, pause_reason=repeated_failures and dispatch one probe per cooldown. A successful probe resumes via the same helper (ACTIVE), which already refuses when the team has since filled its enabled-scout cap.
• Your "must not look like an operator action" requirement is now answered by the schema itself rather than by avoiding enabled: paused_by_system is visibly not an operator's decision, status_changed_by is null on every system transition, and the fleet-facing project profile reports these rows as auto_paused (with the reason) instead of turned_off. enabled does get dual-written — but that's now the enum's derived column, not the operator-intent bit; operator intent lives in paused_by_user, which this breaker can never set or clear.

The mapping should mostly delete code. Happy to help with the rebase.

[posthog]

🦔 ReviewHog reviewed this pull request

Found 0 must fix, 0 should fix, 3 consider.

Published 3 findings (view the review).

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 4dce239b6d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[posthog]

ReviewHog Alpha 🦔 If you find any issues helpful - please reply "valid", "invalid", etc., for evaluation purposes 🙏

[posthog]

ReviewHog Report

Data model

Issues: 1 issue

Files (6)

• products/signals/backend/models.py
• products/signals/backend/migrations/0079_signalscoutconfig_consecutive_failure_count.py
• products/signals/backend/migrations/0080_reconcile_scout_status_enabled.py
• products/signals/backend/migrations/0081_scout_status_constraints.py
• products/signals/backend/migrations/0082_validate_scout_status_constraints.py
• posthog/models/activity_logging/activity_log.py

What were the main changes

• Adds consecutive_failure_count field to SignalScoutConfig, bumped on failure and zeroed on success/resume
• transition_status_by_system now resets the failure streak whenever it transitions a lane to active, closing a stale-streak re-trip bug
• Migration 0080 re-runs the enabled/status reconciliation over rows drifted during the prior rollout window
• Migrations 0081/0082 add-then-validate the deferred CHECK constraints tying enabled to status and pause_reason to the pausing statuses (NOT VALID + ValidateConstraint)
• Activity log exclusions updated so consecutive_failure_count bookkeeping writes don't spam the audit log, while a breaker trip still logs via status like any pause

Feature

Issues: 2 issues

Files (10)

• products/signals/backend/scout_harness/runner.py
• products/signals/backend/temporal/agentic/scout_coordinator.py
• products/signals/backend/scout_harness/limits.py
• products/signals/backend/scout_harness/serializers.py
• products/signals/backend/scout_harness/views.py
• products/signals/backend/scout_harness/AGENTS.md
• frontend/src/scenes/inbox/components/config/scouts/ScoutBadges.tsx
• frontend/src/scenes/inbox/components/config/scouts/ScoutRowCard.tsx
• products/signals/frontend/generated/api.schemas.ts
• services/mcp/src/api/generated.ts

What were the main changes

• New failure-streak circuit breaker: runner's _record_failure_streak/_clear_failure_streak bump/zero the streak and trip/lift a system pause via transition_status_by_system at FAILURE_STREAK_PAUSE_THRESHOLD
• Emits signals_scout_config_auto_paused once per wedge when the breaker trips
• New _collect_probe_runs in the coordinator dispatches one half-open probe per AUTO_PAUSE_PROBE_INTERVAL_S to lanes paused with repeated_failures, cooldown anchored on last_run_at
• New constants FAILURE_STREAK_PAUSE_THRESHOLD and AUTO_PAUSE_PROBE_INTERVAL_S in limits.py
• consecutive_failure_count exposed read-only on the config serializer; resuming via the status endpoint and editing the config both reset the streak (flagged by review comment as incomplete for pre-trip streaks)
• New ScoutPausedBadge shown on the fleet list keyed on status/pause_reason, with a tooltip explaining retry cadence (flagged by review comment as inaccurate for non-daily schedules)
• Generated OpenAPI clients updated for the new consecutive_failure_count field
• AGENTS.md documents the breaker and probe behavior for future maintainers

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 19786b379b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Anchor probe cooldowns to manual breaker trips

When five on-demand runs fail, the shared runner trips this breaker, but start_manual_signals_scout_run() deliberately leaves last_run_at unchanged. For a new config with last_run_at=None, or one whose last scheduled run is already old, this fallback therefore makes the coordinator dispatch another automatic probe on its next tick instead of waiting the full cooldown after the trip. Anchor the initial cooldown to the pause transition time as well, or exclude manual failures from this breaker.

AGENTS.md reference: products/signals/backend/scout_harness/AGENTS.md:L70-L70

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Reset the streak on enabled-only resumes

When an operator resumes a breaker-paused scout through Django admin, the writable enabled checkbox uses SignalScoutConfig.save() to derive status=active, rather than this system-transition helper or the API serializer. That path leaves consecutive_failure_count at the threshold, so the first failed run after the admin resume increments the stale count and immediately pauses the scout again instead of granting five fresh attempts. Apply the same reset when an enabled-only model save resumes a paused row.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Remove the test-class docstring

This newly added class-level docstring violates the repository's explicit rule against doc comments in Python tests. The test names and individual case comments already carry the relevant intent, so remove the docstring rather than adding a second explanation that can drift from the cases.

AGENTS.md reference: AGENTS.md:L186-L186

Useful? React with 👍 / 👎.

[andrewm4894]

/trunk merge

[deployment-status-posthog]

Deploy status

Environment	Status	Deployed At	Workflow
dev	✅ Deployed	2026-07-31 11:45 UTC	Run
prod-us	✅ Deployed	2026-07-31 12:05 UTC	Run
prod-eu	✅ Deployed	2026-07-31 12:08 UTC	Run