Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
80 changes: 61 additions & 19 deletions frontend/src/scenes/inbox/components/config/scouts/ScoutBadges.tsx
Original file line number Diff line number Diff line change
@@ -1,27 +1,69 @@
import { LemonTag, Tooltip } from '@posthog/lemon-ui'

import type { ScoutOriginEnumApi } from 'products/signals/frontend/generated/api.schemas'
import { dayjs } from 'lib/dayjs'

import type {
ScoutOriginEnumApi,
SignalScoutConfigApi as SignalScoutConfig,
} from 'products/signals/frontend/generated/api.schemas'

/**
* Shown when a scout paused itself after failing repeatedly (`status=paused_by_system`,
* `pause_reason=repeated_failures`). Without it the pause is only visible in the API
* response, and a scout that has silently stopped looks the same as one a person turned off.
* Where the scout stands with the system writers that can pause it: the failure breaker
* (`repeated_failures`) or the inactivity sweep (`no_output` / `ignored`), plus the sweep's
* warning state (`pending_pause`). Without it a scout that has silently stopped looks the same
* as one a person turned off. Nothing renders for a healthy scout or a user pause.
*/
export function ScoutPausedBadge(): JSX.Element {
return (
<Tooltip
title={
<span>
This scout paused itself because its last few runs all failed. It retries about once a day and
resumes its normal schedule on the first successful run. Turn it on to resume it right away.
</span>
}
>
<LemonTag type="warning" size="small">
Paused
</LemonTag>
</Tooltip>
)
export function ScoutLifecycleBadge({ config }: { config: SignalScoutConfig }): JSX.Element | null {
if (config.status === 'paused_by_system') {
if (config.pause_reason === 'repeated_failures') {
return (
<Tooltip
title={
<span>
This scout paused itself because its last few runs all failed. It retries about once a day
and resumes its normal schedule on the first successful run. Turn it on to resume it right
away.
</span>
}
>
<LemonTag type="warning" size="small">
Paused
</LemonTag>
</Tooltip>
)
}
if (config.pause_reason === 'no_output' || config.pause_reason === 'ignored') {
const pausedOn = config.status_changed_at
? `Paused on ${dayjs(config.status_changed_at).format('MMMM D, YYYY')}`
: 'Paused'
const why =
config.pause_reason === 'ignored'
? 'because nothing came of its recent reports'
: 'after two weeks without surfacing anything'
return (
<Tooltip title={`${pausedOn} ${why}. Switch it back on to resume it.`}>
<LemonTag type="warning" size="small">
Paused
</LemonTag>
</Tooltip>
)
}
return null
}
if (config.status === 'pending_pause') {
const why =
config.pause_reason === 'ignored'
? "Nothing has come of this scout's recent reports, so it pauses in about a week unless that changes."
: "It hasn't surfaced anything in the last two weeks, so it pauses in about a week unless it finds something."
return (
<Tooltip title={`${why} Turn on 'Keep running while quiet' in its settings to leave it running.`}>
<LemonTag type="caution" size="small">
Quiet
</LemonTag>
</Tooltip>
)
}
return null
}

/** Canonical (PostHog-maintained) vs Custom (team-authored) scout badge. */
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,8 @@ const config: SignalScoutConfigApi = {
output_destinations: {},
last_run_at: null,
consecutive_failure_count: 0,
status_changed_at: null,
auto_pause_exempt: false,
created_at: '2026-07-21T12:00:00Z',
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -135,6 +135,22 @@ export function ScoutConfigForm({
/>
</div>
) : null}
<div className="flex items-center justify-between gap-4">
<div className="flex flex-col min-w-0">
<span className="text-xs text-default">Keep running while quiet</span>
<span className="text-[11.5px] text-muted">
A scout that goes two weeks without surfacing anything anyone uses is paused. Turn this on for a
scout whose job is to stay quiet.
</span>
</div>
<LemonSwitch
size="small"
checked={config.auto_pause_exempt}
disabledReason={controlsDisabledReason}
onChange={(checked) => onUpdate(config.id, { auto_pause_exempt: checked })}
aria-label={`${config.skill_name} keep running while quiet`}
/>
</div>
<ScoutSlackDestination
destination={config.output_destinations?.slack}
onChange={(outputDestinations) => onUpdate(config.id, { output_destinations: outputDestinations })}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ import {
ScoutRollup,
} from '../../../utils/scoutRunsWindow'
import { agentSetupModalLogic } from '../../shell/agentSetupModalLogic'
import { ScoutOriginBadge, ScoutPausedBadge } from './ScoutBadges'
import { ScoutLifecycleBadge, ScoutOriginBadge } from './ScoutBadges'
import { ScoutConfigForm, ScoutEnabledSwitch } from './ScoutConfigControls'
import { ScoutRunBoxes } from './ScoutRunBoxes'

Expand Down Expand Up @@ -129,9 +129,7 @@ export function ScoutRowCard({
</Link>
</Tooltip>
<ScoutOriginBadge origin={config.scout_origin} />
{config.status === 'paused_by_system' && config.pause_reason === 'repeated_failures' ? (
<ScoutPausedBadge />
) : null}
<ScoutLifecycleBadge config={config} />
</div>
</div>
<div className="flex items-center gap-1 whitespace-nowrap text-[11px] text-muted">
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,8 @@ const CREATED_SCOUT: SignalScoutCreateResponseApi = {
output_destinations: {},
last_run_at: null,
consecutive_failure_count: 0,
status_changed_at: null,
auto_pause_exempt: false,
created_at: '2026-07-24T00:00:00Z',
},
}
Expand Down
2 changes: 2 additions & 0 deletions frontend/src/scenes/inbox/logics/scoutFleetLogic.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,8 @@ const BASE_CONFIG: SignalScoutConfigApi = {
output_destinations: {},
last_run_at: null,
consecutive_failure_count: 0,
status_changed_at: null,
auto_pause_exempt: false,
created_at: '2026-07-22T00:00:00Z',
}

Expand Down
1 change: 1 addition & 0 deletions posthog/models/activity_logging/activity_log.py
Original file line number Diff line number Diff line change
Expand Up @@ -336,6 +336,7 @@ class Meta:
"run_interval_minutes": "run interval (minutes)",
"emit": "emit findings",
"pause_reason": "pause reason",
"auto_pause_exempt": "never pause for inactivity",
},
"OAuthApplication": {
"_provisioning_config": "provisioning config",
Expand Down
13 changes: 12 additions & 1 deletion posthog/tasks/scheduled.py
Original file line number Diff line number Diff line change
Expand Up @@ -97,7 +97,11 @@
from products.logs.backend.facade.tasks import logs_alert_events_cleanup_task
from products.pulse.backend.tasks import mark_stale_pulse_briefs_failed
from products.reminders.backend.tasks import process_due_reminders
from products.signals.backend.tasks import refresh_signal_repository_activity, sync_pending_signals_refund_credits
from products.signals.backend.tasks import (
pause_inactive_signal_scouts,
refresh_signal_repository_activity,
sync_pending_signals_refund_credits,
)
from products.stamphog.backend.facade.tasks import DAILY_DIGEST_CRONTAB, send_daily_digests
from products.streamlit_apps.backend.facade.api import (
auto_restart_crashed_streamlit_sandboxes,
Expand Down Expand Up @@ -311,6 +315,13 @@ def setup_periodic_tasks(sender: Celery, **kwargs: Any) -> None:
name="sync pending signals refund credits",
)

# Warn, then pause signals scouts that produce nothing anyone uses - daily at 6:15 AM
sender.add_periodic_task(
crontab(hour="6", minute="15"),
pause_inactive_signal_scouts.s(),
name="pause inactive signals scouts",
)

# Keep the signals repository area-activity cache warm - weekly, Monday early morning
sender.add_periodic_task(
crontab(day_of_week="mon", hour="5", minute="35"),
Expand Down
1 change: 1 addition & 0 deletions products/signals/ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -547,6 +547,7 @@ Per-scout binding for the headless **Signals agent**: one row per `(team, skill_
| `pause_reason` | Char (nullable) | Why the system paused or warned: `no_output` / `ignored` / `repeated_failures`. Also names the writer that owns the pause: a system writer may only clear or overwrite a pause carrying its own reason (`transition_status_by_system()`). Null outside `pending_pause` / `paused_by_system`. |
| `status_changed_at` | DateTime (nullable) | When `status` last changed. Anchors the cold-start grace window (`in_cold_start_grace()`): any move back to `active` re-anchors it, deliberately independent of the actor so the window survives account deletion. |
| `status_changed_by` | FK → User (nullable) | The human behind the last status change; null on system transitions, so a human re-enable and a system resume are distinguishable and a system pause never reads as a person's action. |
| `auto_pause_exempt` | Boolean | Opt-out from the daily inactivity sweep (`scout_harness/inactivity.py`, the `no_output` / `ignored` writer), for watchdog scouts whose whole job is staying quiet. Defaults `False`. Also set automatically when a human re-enables a scout the sweep paused, so the sweep never overrules a person twice. |
| `emit` | Boolean | Dry-run vs emit. Defaults `True`: a freshly authored scout is live from its first tick. Flip to `False` for dry-run — the scout runs and logs but `emit_finding` writes nothing — to validate it on a team before its findings reach the inbox. |
| `run_interval_minutes` | PositiveInteger | Minutes between runs. The coordinator dispatches rolling schedules when `last_run_at is None or now - last_run_at >= run_interval_minutes`. Default `1440` (daily). Validated `30 <= N <= 43200`. |
| `run_cron_schedule` | Char (nullable) | Optional five-field cron expression anchoring runs to wall-clock slots; takes precedence over `run_interval_minutes` when set. Interpreted in the project's timezone so daylight-saving changes are applied automatically. Occurrences must be ≥ 30 minutes apart. Null keeps the rolling interval behavior. |
Expand Down
4 changes: 4 additions & 0 deletions products/signals/backend/activity_logging.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
changes_between,
log_activity,
)
from posthog.models.activity_logging.model_activity import get_current_trigger
from posthog.models.signals import model_activity_signal, mutable_receiver
from posthog.models.user import User

Expand Down Expand Up @@ -57,6 +58,9 @@ def handle_signal_scout_config_change(
detail=Detail(
changes=changes_between(scope, previous=before_update, current=after_update),
name=instance.skill_name,
# Set by system-driven saves (the inactivity sweep) so an entry with no user reads as
# "this job did it" rather than as an unattributed edit.
trigger=get_current_trigger(),
context=SignalScoutConfigContext(skill_name=instance.skill_name),
),
)
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
from django.db import migrations, models


class Migration(migrations.Migration):
dependencies = [
("signals", "0082_validate_scout_status_constraints"),
]

operations = [
migrations.AddField(
model_name="signalscoutconfig",
name="auto_pause_exempt",
field=models.BooleanField(db_default=False, default=False),
),
]
2 changes: 1 addition & 1 deletion products/signals/backend/migrations/max_migration.txt
Original file line number Diff line number Diff line change
@@ -1 +1 @@
0082_validate_scout_status_constraints
0083_signalscoutconfig_auto_pause_exempt
13 changes: 13 additions & 0 deletions products/signals/backend/models.py
Original file line number Diff line number Diff line change
Expand Up @@ -1129,6 +1129,12 @@ class PauseReason(models.TextChoices):
# scheduled by the coordinator. `pending_pause` still runs; the warning is not a pause.
RUNNABLE_STATUSES = (Status.ACTIVE, Status.PENDING_PAUSE)

# The pause reasons the inactivity sweep owns (`scout_harness/inactivity.py`): `no_output`
# for a scout that surfaced nothing, `ignored` for one whose output nobody picked up. On the
# model rather than the sweep because the update serializer also reads them — a human
# re-enable of a pause carrying one of these marks the scout `auto_pause_exempt`.
INACTIVITY_PAUSE_REASONS = (PauseReason.NO_OUTPUT, PauseReason.IGNORED)

# How long a scout is treated as provisional after creation or a human re-enable, during
# which system writers should leave it alone (`in_cold_start_grace`).
COLD_START_GRACE = timedelta(days=14)
Expand Down Expand Up @@ -1196,6 +1202,13 @@ class PauseReason(models.TextChoices):
db_constraint=False,
db_index=False,
)
# Opt-out from the inactivity sweep (`scout_harness/inactivity.py`) — a watchdog whose whole
# value is staying quiet (health checks, inbox validation) is *supposed* to surface nothing
# most weeks, so silence must never read as waste. Also set by the update serializer when a
# human re-enables a scout the sweep paused: that re-enable is a human overruling the rule,
# and the sweep must not undo it one window later. `db_default` alongside `default` keeps the
# AddField non-blocking and the column populated for writers that don't know about it yet.
auto_pause_exempt = models.BooleanField(default=False, db_default=False)
Comment thread
andrewm4894 marked this conversation as resolved.
# Dry-run vs emit. Defaults emit-on so a freshly authored scout is live from its first
# tick. Flip to False for dry-run — the scout runs and logs but `emit_finding` writes
# nothing — to validate it on a team before its findings reach the inbox.
Expand Down
5 changes: 5 additions & 0 deletions products/signals/backend/scout_harness/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,11 @@ In production it is driven by `SignalsScoutCoordinatorWorkflow` (periodic tick e
- `builders.py` — deterministic builders that compute the inventory payload for `SignalProjectProfile`. Sections fall into three layers: capability / configured (sticky — `products_in_use`, `integrations`, `external_data_sources`, `signal_source_configs`, `scout_fleet`, …), aggregated recency (`recent_activity` — per-scope counts off the activity log, cross-cutting orientation across every entity type), and per-entity recent inventory (`recent_surveys`, `recent_feature_flags`, `recent_experiments`, `recent_alerts`, `recent_hog_functions`, `recent_hog_flows`, `recent_notebooks`, `recent_cohorts`, `recent_actions`, `recent_dashboards`, `business_knowledge`).
Per-entity sections are deliberately light (counts + 5 most-recent items with name, status, timestamp); deep drilldowns go via the per-entity MCP list tools.
See the module docstring at `profile/builders.py` for the authoritative section list — when adding or renaming a section, bump `INVENTORY_SOURCE_VERSION` so the cache invalidates cleanly.
- `inactivity.py`
The stop switch on the waste axis (the failure breaker below covers the failure axis). `sweep_inactive_scouts()` warns, then auto-pauses any scout that goes `INACTIVITY_WINDOW` without either producing output on **any** of the three emit channels (`emitted_finding_ids` / `emitted_report_ids` / `edited_report_ids` — the finding tally alone would read every report-channel scout as silent) or having a **person** engage with a report it wrote earlier (a log artefact attributed to a user, or the report reaching a user-driven status — pipeline writers append log artefacts of their own, so unattributed ones don't count).
It speaks the scout lifecycle vocabulary rather than keeping fields of its own: the warning is `status=pending_pause` (still scheduled), the pause is `status=paused_by_system` (syncing `enabled=False`), both through `transition_status_by_system` with a `pause_reason` the sweep owns — `no_output` for a scout that surfaces nothing, `ignored` for one whose reports nobody picks up, separated because the two want different fixes — and the helper's reason-scoped ownership rule keeps the sweep and the failure breaker off each other's pauses.
Driven by the daily `pause_inactive_signal_scouts` Celery task rather than the coordinator tick, which stays short-lived and bounded. A dry-run scout, one inside `in_cold_start_grace()`, one that has barely run (`MIN_RUNS_IN_WINDOW`), and one flagged `auto_pause_exempt` (watchdogs whose value is staying quiet) are all left alone; new warnings are capped per sweep (`MAX_WARNS_PER_SWEEP`), which also bounds later pauses, since a pause only ever follows a warning by `WARNING_GRACE`.
Unlike the failure breaker there is no probe — an inactivity pause never runs again on its own. Re-enabling through the config API resumes the scout, re-anchors its grace window, and marks it `auto_pause_exempt` so the sweep never overrules a person twice; that resume also emits `signals_scout_auto_pause_reverted`, the sweep's false-positive metric (the warn and pause emit `signals_scout_auto_pause_warned` / `signals_scout_auto_paused`).
- `limits.py`
Runtime ceilings as module constants: `DEFAULT_MAX_RUNTIME_S` (per-run budget), `ACTIVITY_SLACK_S`, and `WORKFLOW_HARD_CEILING_S` (`= DEFAULT_MAX_RUNTIME_S + ACTIVITY_SLACK_S`, the activity-level ceiling that gates the workflow's `start_to_close_timeout`).
Also the failure-streak circuit breaker's two knobs: `FAILURE_STREAK_PAUSE_THRESHOLD` (consecutive failed runs before the breaker pauses the lane) and `AUTO_PAUSE_PROBE_INTERVAL_S` (how long a paused lane holds before the coordinator dispatches one probe).
Expand Down
Loading
Loading