Skip to content

Whisper pool, DB contention, differential keep override (2.96.5) - #722

Open
ttlequals0 wants to merge 82 commits into
mainfrom
fix/whisper-prompt-coverage
Open

Whisper pool, DB contention, differential keep override (2.96.5)#722
ttlequals0 wants to merge 82 commits into
mainfrom
fix/whisper-prompt-coverage

Conversation

@ttlequals0

@ttlequals0 ttlequals0 commented Sep 6, 2026

Copy link
Copy Markdown
Owner

2.96.5: database contention and a keep-map hole

Feed discovery wrote a whole feed's episodes inside one transaction holding SQLite's single write lock. On a large archive feed that ran to 80 seconds. Every other writer waits on a 30 second timeout, and past that it fails rather than queues. An RSS sweep, pattern-match recording and podping stamps all reported "database is locked", and one episode lost 15 minutes of processing. Discovery now writes in chunks and takes its existing-episode snapshot outside the write path.

Two smaller contention fixes: an LLM call bumped three global counters as three statements inside its transaction, now one; and a refresh committed once per feed even when the guarded update matched no row.

A marker whose category resolves to keep is now cut when its span overlaps a measured cross-fetch differential region (#728). cross_promo covers both a guest plugging their own show and a paid injected ad for another podcast. The category cannot separate them. The differential can, since injected audio differs between two fetches and host content does not. Both passes apply it, and a toggle under Ad Detection turns it off.

Log records no longer split across lines. A logged prompt body carries newlines, and each became its own log line with no level prefix, so a scraper re-sniffed the level and filed a line reading "CRITICAL:" as a critical entry.

Five releases batched into one PR. Detail in CHANGELOG.md.

2.96.0 to 2.96.1: transcript coverage

Transcripts were losing 7-17% of each episode's speech. Whisper was seeded with a 312-character sponsor-vocabulary initial_prompt, and faster-whisper's batched pipeline prepends it to every VAD clip with no decode fallback. On roughly one clip in ten, large-v3 answered the prompt instead of the audio, and the rest of that clip was dropped. The prompt and its echo scrubber are gone on both backends. Coverage rose from 83-93% to 95.6-96.0%.

Also: the batch-size ceiling no longer sticks after a transient CUDA error; the recents feed (#721), an opt-in combined feed at /recents; publish dates stored as true UTC; chapters listed in episode descriptions (#720); an episode Download menu.

2.96.2: Whisper pool and rate-limit hold

The hold parked each 429'd episode as deferred and leaned on a maintenance pass to return it. The gate was also skipped by user requests and by API-started runs. A 429 now returns the episode to pending in place, keeping its priority, and the gate covers every entry point.

The Whisper pool is opt-in and off by default. Against a remote backend that accepts parallel requests, it runs several episodes at once under a single cap on in-flight requests. Both dials are settings.

2.96.3: stats counted reprocesses twice

The dashboard and per-podcast stats summed a join, so a reprocessed episode's time saved counted once per run (#727). Length, time saved, and episode count now come from the episode's current row. Per-run costs still count every run. The lifetime counter credits only the change since it was last credited, with a backfill.

2.96.4: probes

A rate-limit hold re-probes instead of waiting out a reset that may no longer be true, through llmUsageUrl or a single minimal completion.

A Whisper backend's optional health endpoint is sampled for replica count, model, and total concurrency, with a suggested cap. It is bounded: only while the pool is active, cached 120 seconds, a 15 second sampling budget, one probe per backend. When sampling cannot prove it wrapped the replica set, the UI says "at least N".

Version

2.96.5

Test plan

  • pytest tests/ (6827 passed), ruff clean
  • tsc --noEmit, npm run lint, production build, vitest (832 passed)
  • CodeQL clean on actions, javascript-typescript, and python
  • GPU image scanned (0 HIGH, 0 CRITICAL) and pushed; CPU workflow green on both arches
  • Deployed; /api/v1/health reports 2.96.4; the health probe resolves a three-replica backend with no mismatch

ttlequals0 and others added 24 commits September 6, 2026 18:14
…lists in descriptions, audio downloads (2.96.0)

Transcripts lost 7-17% of each episode because the sponsor-vocabulary
initial_prompt made large-v3 answer the prompt instead of the audio on
roughly one clip in ten; the batched pipeline has no decode fallback.
Remove the prompt and its echo scrubber on both backends.

Also: the batch-size ceiling only lowers on a real OOM, carries a
timestamp, and probes one size up after two days; Queue Held fires once
per pause; OpenCode Go/Zen get the x-opencode-session header (#719);
episode page Download menu with a session-auth processed.mp3 route and
?download=1 attachments; generated chapters can be listed in served
descriptions with a global toggle and per-feed override (#720).
Bumps [gunicorn](https://github.com/benoitc/gunicorn) from 26.1.0 to 26.2.0.
- [Release notes](https://github.com/benoitc/gunicorn/releases)
- [Commits](benoitc/gunicorn@26.1.0...26.2.0)

---
updated-dependencies:
- dependency-name: gunicorn
  dependency-version: 26.2.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Bumps [rapidfuzz](https://github.com/rapidfuzz/RapidFuzz) from 3.14.5 to 3.14.6.
- [Release notes](https://github.com/rapidfuzz/RapidFuzz/releases)
- [Changelog](https://github.com/rapidfuzz/RapidFuzz/blob/main/CHANGELOG.rst)
- [Commits](rapidfuzz/RapidFuzz@v3.14.5...v3.14.6)

---
updated-dependencies:
- dependency-name: rapidfuzz
  dependency-version: 3.14.6
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Bumps [huggingface-hub](https://github.com/huggingface/huggingface_hub) from 1.28.0 to 1.30.0.
- [Release notes](https://github.com/huggingface/huggingface_hub/releases)
- [Commits](huggingface/huggingface_hub@v1.28.0...v1.30.0)

---
updated-dependencies:
- dependency-name: huggingface-hub
  dependency-version: 1.30.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Bumps [cryptography](https://github.com/pyca/cryptography) from 50.0.0 to 50.0.1.
- [Changelog](https://github.com/pyca/cryptography/blob/main/CHANGELOG.rst)
- [Commits](pyca/cryptography@50.0.0...50.0.1)

---
updated-dependencies:
- dependency-name: cryptography
  dependency-version: 50.0.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Bumps [@tanstack/react-query](https://github.com/TanStack/query/tree/HEAD/packages/react-query) from 5.101.4 to 5.102.8.
- [Release notes](https://github.com/TanStack/query/releases)
- [Changelog](https://github.com/TanStack/query/blob/main/packages/react-query/CHANGELOG.md)
- [Commits](https://github.com/TanStack/query/commits/@tanstack/react-query@5.102.8/packages/react-query)

---
updated-dependencies:
- dependency-name: "@tanstack/react-query"
  dependency-version: 5.102.8
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Bumps [@testing-library/user-event](https://github.com/testing-library/user-event) from 14.6.4 to 14.6.6.
- [Release notes](https://github.com/testing-library/user-event/releases)
- [Changelog](https://github.com/testing-library/user-event/blob/main/CHANGELOG.md)
- [Commits](testing-library/user-event@v14.6.4...v14.6.6)

---
updated-dependencies:
- dependency-name: "@testing-library/user-event"
  dependency-version: 14.6.6
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Bumps [swagger-ui-dist](https://github.com/swagger-api/swagger-ui) from 5.32.13 to 5.32.14.
- [Release notes](https://github.com/swagger-api/swagger-ui/releases)
- [Commits](swagger-api/swagger-ui@v5.32.13...v5.32.14)

---
updated-dependencies:
- dependency-name: swagger-ui-dist
  dependency-version: 5.32.14
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Bumps [vitest](https://github.com/vitest-dev/vitest/tree/HEAD/packages/vitest) from 4.1.10 to 4.1.11.
- [Release notes](https://github.com/vitest-dev/vitest/releases)
- [Changelog](https://github.com/vitest-dev/vitest/blob/main/docs/releases.md)
- [Commits](https://github.com/vitest-dev/vitest/commits/v4.1.11/packages/vitest)

---
updated-dependencies:
- dependency-name: vitest
  dependency-version: 4.1.11
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Bumps [eslint](https://github.com/eslint/eslint) from 10.8.1 to 10.9.1.
- [Release notes](https://github.com/eslint/eslint/releases)
- [Commits](eslint/eslint@v10.8.1...v10.9.1)

---
updated-dependencies:
- dependency-name: eslint
  dependency-version: 10.9.1
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
…ride, stale cache on deletes, has_upstream gate)
@ttlequals0 ttlequals0 changed the title Whisper prompt removal, chapter lists, audio downloads (2.96.0) Recents feed, transcript coverage fix, downloads, chapter notes (2.96.1) Sep 7, 2026
…nts in OPML (2.96.2)

A 429 with a reset now puts the episode back to pending and releases its
claimed queue row in place, so it keeps its priority and position. The
pause is enforced in start_background_processing, so Play and Reprocess
wait with the rest. The stale marker clears on the first loop pass after
the reset. Deferred parking, the TTL, holdCount, rateLimitHeld and the
ttl_hours/requeued webhook fields are gone; a one-time migration returns
episodes an older hold parked to their original queue rows.

Dashboard feed cards and episode rows reserve the same slots so they line
up. The modified OPML export lists the recents feed alongside local feeds.
Add _clamp_seconds in rate_limit.py and _bump_stat on the stats mixin
so the repeated clamp and upsert-with-delta logic lives in one place.
Trim docstrings on credit_time_saved, get_dashboard_stats,
get_stats_by_podcast, and extract_retry_after that repeated rationale
already stated elsewhere.
…tats floor (2.96.3)

- Backfill episodes.credited_time_saved on upgrade so the pre-existing
  double-credit is corrected and the lifetime total matches the dashboard.
- Retry the tail Whisper upload without vad_filter when a server 400s the
  unrecognized field, instead of losing the tail outright.
- credit_time_saved runs unconditionally so a recut that removes a saving
  corrects the counter downward.
- _bump_stat floors a first negative write at zero, not just an update.
- parse_upstream_reset now descends into a proxied error's metadata.raw,
  matching the Google coercer.
- Relabel run-scoped token averages "Avg Tokens/Run" on the stats page and
  show the episode count backing "Total Time Saved".
- Add credited_time_saved to the episodes DDL; dedupe test fixtures and
  helper duplication.
…ng it out

A held queue previously slept until the 429's own stated reset, which can
be wrong in either direction: too short against the provider's real reset,
or stale once a plan change or added credit clears the limit early.
probe_rate_limit() now runs periodically while a hold is active: an
operator-configured usage endpoint (llmUsageUrl) is checked first and can
clear or re-stamp the hold with a fresher reset in either direction; without
one, a single minimal completion call does the same check. New settings
llmUsageUrl and rateLimitProbeMinutes (0 disables) are surfaced on
/settings/rate-limit-hold. The failure handler also prefers the usage
endpoint's reset over the 429's own value when stamping the first hold.
Usage endpoint URL and probe interval inputs in the rate-limit hold block,
reusing the existing hold-block field layout and NumberInput component.
The rate-limit hold probe (86639f0, 4dffc85, 769f6f5) landed after
2.96.3 was already built and deployed from a7fe518, but its changelog
entry was written into the 2.96.3 section. Move it into a new 2.96.4
section and bump version.py and openapi.yaml to match.
- CHANGELOG: note the vad_filter rejection retry, not just servers that
  silently ignore the field.
- Guard the credited_time_saved backfill's INFO log on rowcount, so a
  fresh install does not log a zero-row backfill.
- Move the Database singleton reset in the credited_time_saved migration
  test to fixture teardown, so an assertion failure cannot leak it into
  later tests.
- _coerce_error_dict: only descend into error.metadata.raw when the outer
  error has no reset field of its own, so an outer seconds_until_reset is
  not shadowed by a proxied inner body.
Sample a self-hosted Whisper backend's optional /health endpoint to
report per-instance model, device, compute type, and concurrency, and
suggest a whisperPoolMaxRequests value from the combined concurrency.
Behind a load balancer, repeated calls round-robin across replicas, so
sampling reveals the replica count without operator guesswork.

transcriber.probe_whisper_health() is wired into GET
/settings/whisper/capacity and the Whisper connection test; both
degrade to an unavailable result on any probe failure without touching
the rest of the payload. Documented in openapi.yaml and
docs/configuration.md.
Add the health probe shape to WhisperCapacity and ConnectionTestResult,
and show it under the Whisper pool's existing capacity line: instance
count, model, and a suggested cap when it differs from what is
configured, or a warning when instances disagree on model, device, or
compute type.
An outer seconds_until_reset, resets_at, or resets_at_iso that was empty,
non-numeric, or explicitly null still counted as present, so the descent
into a proxied body under error.metadata.raw was skipped and the parse
fell back to the generic Retry-After header. Presence is now decided by
whether the field actually parses, shared with parse_upstream_reset.
The probe fired on every 15s capacity poll, up to three outbound requests
per tick, including on instances that had switched back to the local
backend and left a stale URL behind. It now runs only while the pool is
active and caches per base URL for 120s; the connection test passes
use_cache=False so it still checks the backend on demand.

Sampling was capped at 3 requests against a function default of 5, so a
backend with more replicas could never report them all, and two repeated
instances were enough to stop early on a load balancer that is not
strictly round robin. The default is now 8 with a three-repeat stop, and
the result carries sampled_floor so the count can be presented as a lower
bound when the sampling never wrapped around.

Also: send the API key so a backend requiring auth on /health is not
reported unavailable, floor each instance's max_concurrent at 1 so a
backend reporting 0 cannot suggest a cap below its own replica count,
coerce list and dict health fields so an odd payload cannot raise, and
stop the capacity endpoint tests from making real outbound requests.
The health line stated the detected instance count as fact even when
every sample had turned up a new replica, which only establishes a floor.
It now reads "at least N instances" in that case. Mismatch warnings map
field names to readable labels instead of printing compute_type. The
configuration docs note that the probe reads {base}/health, so a server
exposing health at its root reports nothing.
sampled_floor was cleared by any single repeated instance, but one repeat
is not evidence the sampling wrapped the replica set. Behind a balancer
that is not round robin, a run like A B C A D E F B exhausts its samples
without ever repeating three times running, and the old flag presented
that undercount as the confirmed replica total. The flag now follows the
early-stop break, which is the only signal that sampling converged.

max_concurrent reported as a JSON float or a numeric string counted as 1,
so four such replicas suggested a cap of 4 instead of 16 and told the
operator to throttle a healthy fleet. Numeric spellings are now accepted,
with bool and unparseable values still counting as 1.

The connection test claimed a uniform model from instances[0] even when
the probe had recorded a mismatch, contradicting the warning Settings
raises from the same payload, and it ignored sampled_floor while the
panel hedged. Both surfaces now say the same thing.

A failed probe was cached for 120 seconds with no way to clear it, so a
backend that came back up, or gained a health endpoint, stayed invisible.
use_cache is now refresh, which skips the read and stores the fresh
result, so the connection test evicts a stale entry.

Also: the floor hedge leads the Settings line so it governs the request
total, which is summed over the same undercounted instances, and the
capacity comment and changelog now say the probe is gated on the pool
being active rather than on the backend alone.
A max_concurrent of Infinity or "1e400" raised OverflowError out of
int(float(...)), which the except tuple did not catch, so a single odd
health payload could 500 the capacity endpoint and the connection test
despite the probe's never-raises contract. json.loads accepts bare
Infinity, so this needs no malicious backend.

Raising the sample count to 8 made a hanging backend expensive: one probe
could hold a worker for 40 seconds, and the 15 second settings poll would
stack more behind it, since the cache is only written once a probe
returns. A probe now stops sampling after 15 seconds and runs under a
single-flight lock; a caller arriving mid-probe gets the cached result
rather than queuing another.

A refresh no longer caches a failure. The connection test refreshes, and
one transient 429 there should not blank a good probe for two minutes.

The OpenAPI description and changelog still stated the pre-fix rule for
sampled_floor, that no instance ever repeated, which the code no longer
implements. Both now describe the early-stop rule. In Settings the hedge
also reaches the request total, which is summed over the same
undercounted instances and is equally a floor.
The single-flight lock was module-global and keyed on nothing, so a probe
hanging against one backend blocked a connection test against a different
one, and that test then reported a healthy backend as unavailable. The
guard is now an in-flight set keyed by base URL.

A caller that arrives mid-probe was also handed {'available': False},
which blanks the Settings health line. A non-refresh caller only reaches
that path because its own cache read just missed, so the cache is empty by
definition and the line blinked out for the length of every probe past the
120 second TTL. The last successful probe per backend is now retained past
the TTL to answer exactly that case.

The budget caps how many samples a probe starts, not the wall time of one
already in flight, since a redirecting backend can stretch a single
request well past the per-request timeout. The docstring said otherwise.

The budget test drove a patched time.monotonic, which replaces the
attribute process-wide for the duration of the block. An exhausted budget
makes the same point with no clock patching at all.
The last good probe kept per backend never expired, so a caller arriving
while a probe was in flight could be served a result of any age, and the
Settings line gates only on availability. A backend that hangs rather than
refusing would show "N instances reporting large-v3" once per cache cycle
after being dead for hours. Showing a long-dead backend as healthy is a
worse failure than showing nothing, so a retained result is now stamped
and only served within three cache TTLs.

Eviction cleared the whole map, dropping the entry the retention exists to
serve. The connection test takes an arbitrary baseUrl from the request
body, so 32 successful tests against candidate URLs would wipe the
configured backend's entry. It now drops the oldest entry instead.

The changelog claimed a hanging backend could no longer stall the settings
page. The guard is per process, and the request that starts a probe still
waits for it; what it prevents is the polling multiplying that cost. The
schema now also says a result may not be live.

test_a_probe_in_flight_with_nothing_known_reports_unavailable asserted
only the return value, and the probe's own except Exception swallows the
mock's side effect, so it passed with the guard ignored entirely. It now
asserts no request was made.
This is a public repo and its prose rules ban a double hyphen used as a
dash. The SQL comments in the schema are untouched, since those are
syntax rather than prose.
@ttlequals0 ttlequals0 changed the title Whisper pool, rate-limit hold fix, recents feed, transcript coverage (2.96.2) Whisper pool, rate-limit probe, backend health, stats fix (2.96.4) Sep 7, 2026
SQLite allows one writer, so a write transaction's cost is paid by every
other writer queued behind it. Two paths were adding avoidable trips.

record_token_usage bumped three global counters as three separate
statements inside the same transaction as its token_usage upsert. Every
LLM call from every detector window and reviewer thread does this, and
all of them contend on the same three rows. The three are now a single
multi-row upsert.

clear_refresh_failure_state committed unconditionally, including for a
clean feed whose guarded UPDATE matched no row. A refresh sweep runs five
threads over every feed, so most of those commits bought nothing. It now
rolls back instead, which also releases the lock rather than stranding an
open transaction.

Separately, a logged prompt body carries newlines, and a container runtime
turns each into its own log line. The continuation lines have no level
prefix, so a scraper re-sniffs the level from the text and files a line
beginning "CRITICAL:" as a critical entry, which pollutes error views and
would misfire any alert keyed on level. The console formatter now collapses
newlines in the message. It copies the record rather than mutating it,
since the per-episode run log handler formats the same record and wants
the real line breaks; exception tracebacks are appended afterwards and keep
theirs.
A category cannot decide this on its own. cross_promo covers both a guest
plugging their own show, which is why an operator maps it to keep, and a
paid dynamically-inserted ad for another podcast. Mapping it to keep left
real injected ads in the cut, and a contiguous break came out as cut, kept,
cut.

The cross-fetch differential separates the two cleanly: injected audio
differs between two fetches of the same enclosure and host content does
not. A marker overlapping a measured differential region now bypasses keep
the way a defined pattern already does, behind
dai_differential_overrides_keep (default true).

The overlap predicate moves to differential_fetcher as
differential_region_overlapping and the validator's Layer 3 corroboration
now calls it too, so the two readings of the same evidence cannot drift.
The corr gate carries over unchanged: a high-corr region mostly matched
across fetches and proves nothing, and an unmeasured region proves nothing
either.

Reported with the region table and a full archive breakdown that showed no
overlap between the two meanings, which is what made the rule safe to make
standing rather than advisory.
A refresh wrote a whole feed's discovered episodes inside a single
immediate transaction. That holds SQLite's one write lock across a SELECT,
then per-row date parsing, a dedupe lookup, an upsert and index writes for
every episode. On a large archive feed the lock was observed held for 82
seconds.

Every other writer waits on a 30 second busy_timeout, so past that they do
not queue, they fail: "database is locked" across a whole RSS sweep,
pattern-match recording, podping host stamps, and one episode that lost 15
minutes of processing to it. The concurrent pool made this reachable by
adding writers, but the long transaction is what made it fatal rather than
slow.

Discovery now writes in chunks so the lock is released between them, and
takes its existing-episode snapshot outside the write path rather than
holding the lock across that read. The upsert is idempotent, so a chunk
failing part way leaves earlier chunks applied and the next refresh
reconciles.
Consolidate the two parses of episode_details.dai_differential_json onto
one helper; they had drifted to different exception sets in the same file.

Interpolate log args only when a record has them, so the common no-args
record is not built twice per line at DEBUG volume.

Drop the redundant regions check in the keep override, which the overlap
helper already handles, and record why that helper uses a strict overlap
rather than utils.time.ranges_overlap: a marker touching a region's edge
shares no audio with it.

State the lock rule on transaction() itself, where the next bulk writer
will read it, and note that the batched stat bump has no clamp.

One transaction spy in the discovery tests instead of three copies.
Pass 2 honoured only the pattern override, so an injected ad that a kept
category sheltered in pass 1 was routed to category_kept and survived in
the audio, with no later sweep to catch it. It now applies the differential
override too, testing the original marker: the regions are in original-audio
coordinates and the processed marker is not.

The setting had no control. The changelog told operators to set
daiDifferentialOverridesKeep to false while the only way to do that was a
raw settings PATCH, so a default-on behaviour change had no off switch in
the UI. Adds a toggle beside the other differential settings.

Also corrects three comments that claimed more than the code delivers:
index_episodes swallows its exceptions when the caller owns the
transaction, so chunked indexing is not the guarantee the comment asserted;
discovery is no longer atomic across a feed; and the existing-GUID snapshot
now read outside the write lock lets a concurrent refresh of the same feed
inflate the new-episode count.

Not applied: the report's TypeError on comparing corr against a string
default. registry_get_default coerces by payload_kind and returns a float
for this key, so the comparison is float to float.
@ttlequals0 ttlequals0 changed the title Whisper pool, rate-limit probe, backend health, stats fix (2.96.4) Whisper pool, DB contention, differential keep override (2.96.5) Sep 8, 2026
The registry entry alone does not reach the API. GET /settings and the
ad-detection PUT are assembled from hand-written lists, so the new key
passed every registry test while being absent from the response and
unsettable from the UI: the toggle rendered at its useState default and
neither loaded nor saved.

Adds a test that walks SETTINGS_REGISTRY against the GET payload, which is
the check that was missing. It found one other gap on its first run,
notification_timezone, which turns out to be served by its own endpoint and
is listed as a deliberate exclusion with that reason.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant