Whisper pool, DB contention, differential keep override (2.96.5) - #722
Open
ttlequals0 wants to merge 82 commits into
Open
Whisper pool, DB contention, differential keep override (2.96.5)#722ttlequals0 wants to merge 82 commits into
ttlequals0 wants to merge 82 commits into
Conversation
…lists in descriptions, audio downloads (2.96.0) Transcripts lost 7-17% of each episode because the sponsor-vocabulary initial_prompt made large-v3 answer the prompt instead of the audio on roughly one clip in ten; the batched pipeline has no decode fallback. Remove the prompt and its echo scrubber on both backends. Also: the batch-size ceiling only lowers on a real OOM, carries a timestamp, and probes one size up after two days; Queue Held fires once per pause; OpenCode Go/Zen get the x-opencode-session header (#719); episode page Download menu with a session-auth processed.mp3 route and ?download=1 attachments; generated chapters can be listed in served descriptions with a global toggle and per-feed override (#720).
…ightward on phones (2.96.1)
Bumps [gunicorn](https://github.com/benoitc/gunicorn) from 26.1.0 to 26.2.0. - [Release notes](https://github.com/benoitc/gunicorn/releases) - [Commits](benoitc/gunicorn@26.1.0...26.2.0) --- updated-dependencies: - dependency-name: gunicorn dependency-version: 26.2.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>
Bumps [rapidfuzz](https://github.com/rapidfuzz/RapidFuzz) from 3.14.5 to 3.14.6. - [Release notes](https://github.com/rapidfuzz/RapidFuzz/releases) - [Changelog](https://github.com/rapidfuzz/RapidFuzz/blob/main/CHANGELOG.rst) - [Commits](rapidfuzz/RapidFuzz@v3.14.5...v3.14.6) --- updated-dependencies: - dependency-name: rapidfuzz dependency-version: 3.14.6 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>
Bumps [huggingface-hub](https://github.com/huggingface/huggingface_hub) from 1.28.0 to 1.30.0. - [Release notes](https://github.com/huggingface/huggingface_hub/releases) - [Commits](huggingface/huggingface_hub@v1.28.0...v1.30.0) --- updated-dependencies: - dependency-name: huggingface-hub dependency-version: 1.30.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>
Bumps [cryptography](https://github.com/pyca/cryptography) from 50.0.0 to 50.0.1. - [Changelog](https://github.com/pyca/cryptography/blob/main/CHANGELOG.rst) - [Commits](pyca/cryptography@50.0.0...50.0.1) --- updated-dependencies: - dependency-name: cryptography dependency-version: 50.0.1 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>
Bumps [@tanstack/react-query](https://github.com/TanStack/query/tree/HEAD/packages/react-query) from 5.101.4 to 5.102.8. - [Release notes](https://github.com/TanStack/query/releases) - [Changelog](https://github.com/TanStack/query/blob/main/packages/react-query/CHANGELOG.md) - [Commits](https://github.com/TanStack/query/commits/@tanstack/react-query@5.102.8/packages/react-query) --- updated-dependencies: - dependency-name: "@tanstack/react-query" dependency-version: 5.102.8 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>
Bumps [@testing-library/user-event](https://github.com/testing-library/user-event) from 14.6.4 to 14.6.6. - [Release notes](https://github.com/testing-library/user-event/releases) - [Changelog](https://github.com/testing-library/user-event/blob/main/CHANGELOG.md) - [Commits](testing-library/user-event@v14.6.4...v14.6.6) --- updated-dependencies: - dependency-name: "@testing-library/user-event" dependency-version: 14.6.6 dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>
Bumps [swagger-ui-dist](https://github.com/swagger-api/swagger-ui) from 5.32.13 to 5.32.14. - [Release notes](https://github.com/swagger-api/swagger-ui/releases) - [Commits](swagger-api/swagger-ui@v5.32.13...v5.32.14) --- updated-dependencies: - dependency-name: swagger-ui-dist dependency-version: 5.32.14 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>
Bumps [vitest](https://github.com/vitest-dev/vitest/tree/HEAD/packages/vitest) from 4.1.10 to 4.1.11. - [Release notes](https://github.com/vitest-dev/vitest/releases) - [Changelog](https://github.com/vitest-dev/vitest/blob/main/docs/releases.md) - [Commits](https://github.com/vitest-dev/vitest/commits/v4.1.11/packages/vitest) --- updated-dependencies: - dependency-name: vitest dependency-version: 4.1.11 dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>
Bumps [eslint](https://github.com/eslint/eslint) from 10.8.1 to 10.9.1. - [Release notes](https://github.com/eslint/eslint/releases) - [Commits](eslint/eslint@v10.8.1...v10.9.1) --- updated-dependencies: - dependency-name: eslint dependency-version: 10.9.1 dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>
…off, row flags, single count)
…c, hidden filters)
…ride, stale cache on deletes, has_upstream gate)
…nts in OPML (2.96.2) A 429 with a reset now puts the episode back to pending and releases its claimed queue row in place, so it keeps its priority and position. The pause is enforced in start_background_processing, so Play and Reprocess wait with the rest. The stale marker clears on the first loop pass after the reset. Deferred parking, the TTL, holdCount, rateLimitHeld and the ttl_hours/requeued webhook fields are gone; a one-time migration returns episodes an older hold parked to their original queue rows. Dashboard feed cards and episode rows reserve the same slots so they line up. The modified OPML export lists the recents feed alongside local feeds.
Add _clamp_seconds in rate_limit.py and _bump_stat on the stats mixin so the repeated clamp and upsert-with-delta logic lives in one place. Trim docstrings on credit_time_saved, get_dashboard_stats, get_stats_by_podcast, and extract_retry_after that repeated rationale already stated elsewhere.
…tats floor (2.96.3) - Backfill episodes.credited_time_saved on upgrade so the pre-existing double-credit is corrected and the lifetime total matches the dashboard. - Retry the tail Whisper upload without vad_filter when a server 400s the unrecognized field, instead of losing the tail outright. - credit_time_saved runs unconditionally so a recut that removes a saving corrects the counter downward. - _bump_stat floors a first negative write at zero, not just an update. - parse_upstream_reset now descends into a proxied error's metadata.raw, matching the Google coercer. - Relabel run-scoped token averages "Avg Tokens/Run" on the stats page and show the episode count backing "Total Time Saved". - Add credited_time_saved to the episodes DDL; dedupe test fixtures and helper duplication.
…ng it out A held queue previously slept until the 429's own stated reset, which can be wrong in either direction: too short against the provider's real reset, or stale once a plan change or added credit clears the limit early. probe_rate_limit() now runs periodically while a hold is active: an operator-configured usage endpoint (llmUsageUrl) is checked first and can clear or re-stamp the hold with a fresher reset in either direction; without one, a single minimal completion call does the same check. New settings llmUsageUrl and rateLimitProbeMinutes (0 disables) are surfaced on /settings/rate-limit-hold. The failure handler also prefers the usage endpoint's reset over the 429's own value when stamping the first hold.
Usage endpoint URL and probe interval inputs in the rate-limit hold block, reusing the existing hold-block field layout and NumberInput component.
- CHANGELOG: note the vad_filter rejection retry, not just servers that silently ignore the field. - Guard the credited_time_saved backfill's INFO log on rowcount, so a fresh install does not log a zero-row backfill. - Move the Database singleton reset in the credited_time_saved migration test to fixture teardown, so an assertion failure cannot leak it into later tests. - _coerce_error_dict: only descend into error.metadata.raw when the outer error has no reset field of its own, so an outer seconds_until_reset is not shadowed by a proxied inner body.
Sample a self-hosted Whisper backend's optional /health endpoint to report per-instance model, device, compute type, and concurrency, and suggest a whisperPoolMaxRequests value from the combined concurrency. Behind a load balancer, repeated calls round-robin across replicas, so sampling reveals the replica count without operator guesswork. transcriber.probe_whisper_health() is wired into GET /settings/whisper/capacity and the Whisper connection test; both degrade to an unavailable result on any probe failure without touching the rest of the payload. Documented in openapi.yaml and docs/configuration.md.
Add the health probe shape to WhisperCapacity and ConnectionTestResult, and show it under the Whisper pool's existing capacity line: instance count, model, and a suggested cap when it differs from what is configured, or a warning when instances disagree on model, device, or compute type.
An outer seconds_until_reset, resets_at, or resets_at_iso that was empty, non-numeric, or explicitly null still counted as present, so the descent into a proxied body under error.metadata.raw was skipped and the parse fell back to the generic Retry-After header. Presence is now decided by whether the field actually parses, shared with parse_upstream_reset.
The probe fired on every 15s capacity poll, up to three outbound requests per tick, including on instances that had switched back to the local backend and left a stale URL behind. It now runs only while the pool is active and caches per base URL for 120s; the connection test passes use_cache=False so it still checks the backend on demand. Sampling was capped at 3 requests against a function default of 5, so a backend with more replicas could never report them all, and two repeated instances were enough to stop early on a load balancer that is not strictly round robin. The default is now 8 with a three-repeat stop, and the result carries sampled_floor so the count can be presented as a lower bound when the sampling never wrapped around. Also: send the API key so a backend requiring auth on /health is not reported unavailable, floor each instance's max_concurrent at 1 so a backend reporting 0 cannot suggest a cap below its own replica count, coerce list and dict health fields so an odd payload cannot raise, and stop the capacity endpoint tests from making real outbound requests.
The health line stated the detected instance count as fact even when
every sample had turned up a new replica, which only establishes a floor.
It now reads "at least N instances" in that case. Mismatch warnings map
field names to readable labels instead of printing compute_type. The
configuration docs note that the probe reads {base}/health, so a server
exposing health at its root reports nothing.
sampled_floor was cleared by any single repeated instance, but one repeat is not evidence the sampling wrapped the replica set. Behind a balancer that is not round robin, a run like A B C A D E F B exhausts its samples without ever repeating three times running, and the old flag presented that undercount as the confirmed replica total. The flag now follows the early-stop break, which is the only signal that sampling converged. max_concurrent reported as a JSON float or a numeric string counted as 1, so four such replicas suggested a cap of 4 instead of 16 and told the operator to throttle a healthy fleet. Numeric spellings are now accepted, with bool and unparseable values still counting as 1. The connection test claimed a uniform model from instances[0] even when the probe had recorded a mismatch, contradicting the warning Settings raises from the same payload, and it ignored sampled_floor while the panel hedged. Both surfaces now say the same thing. A failed probe was cached for 120 seconds with no way to clear it, so a backend that came back up, or gained a health endpoint, stayed invisible. use_cache is now refresh, which skips the read and stores the fresh result, so the connection test evicts a stale entry. Also: the floor hedge leads the Settings line so it governs the request total, which is summed over the same undercounted instances, and the capacity comment and changelog now say the probe is gated on the pool being active rather than on the backend alone.
A max_concurrent of Infinity or "1e400" raised OverflowError out of int(float(...)), which the except tuple did not catch, so a single odd health payload could 500 the capacity endpoint and the connection test despite the probe's never-raises contract. json.loads accepts bare Infinity, so this needs no malicious backend. Raising the sample count to 8 made a hanging backend expensive: one probe could hold a worker for 40 seconds, and the 15 second settings poll would stack more behind it, since the cache is only written once a probe returns. A probe now stops sampling after 15 seconds and runs under a single-flight lock; a caller arriving mid-probe gets the cached result rather than queuing another. A refresh no longer caches a failure. The connection test refreshes, and one transient 429 there should not blank a good probe for two minutes. The OpenAPI description and changelog still stated the pre-fix rule for sampled_floor, that no instance ever repeated, which the code no longer implements. Both now describe the early-stop rule. In Settings the hedge also reaches the request total, which is summed over the same undercounted instances and is equally a floor.
The single-flight lock was module-global and keyed on nothing, so a probe
hanging against one backend blocked a connection test against a different
one, and that test then reported a healthy backend as unavailable. The
guard is now an in-flight set keyed by base URL.
A caller that arrives mid-probe was also handed {'available': False},
which blanks the Settings health line. A non-refresh caller only reaches
that path because its own cache read just missed, so the cache is empty by
definition and the line blinked out for the length of every probe past the
120 second TTL. The last successful probe per backend is now retained past
the TTL to answer exactly that case.
The budget caps how many samples a probe starts, not the wall time of one
already in flight, since a redirecting backend can stretch a single
request well past the per-request timeout. The docstring said otherwise.
The budget test drove a patched time.monotonic, which replaces the
attribute process-wide for the duration of the block. An exhausted budget
makes the same point with no clock patching at all.
The last good probe kept per backend never expired, so a caller arriving while a probe was in flight could be served a result of any age, and the Settings line gates only on availability. A backend that hangs rather than refusing would show "N instances reporting large-v3" once per cache cycle after being dead for hours. Showing a long-dead backend as healthy is a worse failure than showing nothing, so a retained result is now stamped and only served within three cache TTLs. Eviction cleared the whole map, dropping the entry the retention exists to serve. The connection test takes an arbitrary baseUrl from the request body, so 32 successful tests against candidate URLs would wipe the configured backend's entry. It now drops the oldest entry instead. The changelog claimed a hanging backend could no longer stall the settings page. The guard is per process, and the request that starts a probe still waits for it; what it prevents is the polling multiplying that cost. The schema now also says a result may not be live. test_a_probe_in_flight_with_nothing_known_reports_unavailable asserted only the return value, and the probe's own except Exception swallows the mock's side effect, so it passed with the guard ignored entirely. It now asserts no request was made.
This is a public repo and its prose rules ban a double hyphen used as a dash. The SQL comments in the schema are untouched, since those are syntax rather than prose.
SQLite allows one writer, so a write transaction's cost is paid by every other writer queued behind it. Two paths were adding avoidable trips. record_token_usage bumped three global counters as three separate statements inside the same transaction as its token_usage upsert. Every LLM call from every detector window and reviewer thread does this, and all of them contend on the same three rows. The three are now a single multi-row upsert. clear_refresh_failure_state committed unconditionally, including for a clean feed whose guarded UPDATE matched no row. A refresh sweep runs five threads over every feed, so most of those commits bought nothing. It now rolls back instead, which also releases the lock rather than stranding an open transaction. Separately, a logged prompt body carries newlines, and a container runtime turns each into its own log line. The continuation lines have no level prefix, so a scraper re-sniffs the level from the text and files a line beginning "CRITICAL:" as a critical entry, which pollutes error views and would misfire any alert keyed on level. The console formatter now collapses newlines in the message. It copies the record rather than mutating it, since the per-episode run log handler formats the same record and wants the real line breaks; exception tracebacks are appended afterwards and keep theirs.
A category cannot decide this on its own. cross_promo covers both a guest plugging their own show, which is why an operator maps it to keep, and a paid dynamically-inserted ad for another podcast. Mapping it to keep left real injected ads in the cut, and a contiguous break came out as cut, kept, cut. The cross-fetch differential separates the two cleanly: injected audio differs between two fetches of the same enclosure and host content does not. A marker overlapping a measured differential region now bypasses keep the way a defined pattern already does, behind dai_differential_overrides_keep (default true). The overlap predicate moves to differential_fetcher as differential_region_overlapping and the validator's Layer 3 corroboration now calls it too, so the two readings of the same evidence cannot drift. The corr gate carries over unchanged: a high-corr region mostly matched across fetches and proves nothing, and an unmeasured region proves nothing either. Reported with the region table and a full archive breakdown that showed no overlap between the two meanings, which is what made the rule safe to make standing rather than advisory.
A refresh wrote a whole feed's discovered episodes inside a single immediate transaction. That holds SQLite's one write lock across a SELECT, then per-row date parsing, a dedupe lookup, an upsert and index writes for every episode. On a large archive feed the lock was observed held for 82 seconds. Every other writer waits on a 30 second busy_timeout, so past that they do not queue, they fail: "database is locked" across a whole RSS sweep, pattern-match recording, podping host stamps, and one episode that lost 15 minutes of processing to it. The concurrent pool made this reachable by adding writers, but the long transaction is what made it fatal rather than slow. Discovery now writes in chunks so the lock is released between them, and takes its existing-episode snapshot outside the write path rather than holding the lock across that read. The upsert is idempotent, so a chunk failing part way leaves earlier chunks applied and the next refresh reconciles.
Consolidate the two parses of episode_details.dai_differential_json onto one helper; they had drifted to different exception sets in the same file. Interpolate log args only when a record has them, so the common no-args record is not built twice per line at DEBUG volume. Drop the redundant regions check in the keep override, which the overlap helper already handles, and record why that helper uses a strict overlap rather than utils.time.ranges_overlap: a marker touching a region's edge shares no audio with it. State the lock rule on transaction() itself, where the next bulk writer will read it, and note that the batched stat bump has no clamp. One transaction spy in the discovery tests instead of three copies.
Pass 2 honoured only the pattern override, so an injected ad that a kept category sheltered in pass 1 was routed to category_kept and survived in the audio, with no later sweep to catch it. It now applies the differential override too, testing the original marker: the regions are in original-audio coordinates and the processed marker is not. The setting had no control. The changelog told operators to set daiDifferentialOverridesKeep to false while the only way to do that was a raw settings PATCH, so a default-on behaviour change had no off switch in the UI. Adds a toggle beside the other differential settings. Also corrects three comments that claimed more than the code delivers: index_episodes swallows its exceptions when the caller owns the transaction, so chunked indexing is not the guarantee the comment asserted; discovery is no longer atomic across a feed; and the existing-GUID snapshot now read outside the write lock lets a concurrent refresh of the same feed inflate the new-episode count. Not applied: the report's TypeError on comparing corr against a string default. registry_get_default coerces by payload_kind and returns a float for this key, so the comparison is float to float.
The registry entry alone does not reach the API. GET /settings and the ad-detection PUT are assembled from hand-written lists, so the new key passed every registry test while being absent from the response and unsettable from the UI: the toggle rendered at its useState default and neither loaded nor saved. Adds a test that walks SETTINGS_REGISTRY against the GET payload, which is the check that was missing. It found one other gap on its first run, notification_timezone, which turns out to be served by its own endpoint and is listed as a deliberate exclusion with that reason.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
2.96.5: database contention and a keep-map hole
Feed discovery wrote a whole feed's episodes inside one transaction holding SQLite's single write lock. On a large archive feed that ran to 80 seconds. Every other writer waits on a 30 second timeout, and past that it fails rather than queues. An RSS sweep, pattern-match recording and podping stamps all reported "database is locked", and one episode lost 15 minutes of processing. Discovery now writes in chunks and takes its existing-episode snapshot outside the write path.
Two smaller contention fixes: an LLM call bumped three global counters as three statements inside its transaction, now one; and a refresh committed once per feed even when the guarded update matched no row.
A marker whose category resolves to
keepis now cut when its span overlaps a measured cross-fetch differential region (#728).cross_promocovers both a guest plugging their own show and a paid injected ad for another podcast. The category cannot separate them. The differential can, since injected audio differs between two fetches and host content does not. Both passes apply it, and a toggle under Ad Detection turns it off.Log records no longer split across lines. A logged prompt body carries newlines, and each became its own log line with no level prefix, so a scraper re-sniffed the level and filed a line reading "CRITICAL:" as a critical entry.
Five releases batched into one PR. Detail in CHANGELOG.md.
2.96.0 to 2.96.1: transcript coverage
Transcripts were losing 7-17% of each episode's speech. Whisper was seeded with a 312-character sponsor-vocabulary
initial_prompt, and faster-whisper's batched pipeline prepends it to every VAD clip with no decode fallback. On roughly one clip in ten, large-v3 answered the prompt instead of the audio, and the rest of that clip was dropped. The prompt and its echo scrubber are gone on both backends. Coverage rose from 83-93% to 95.6-96.0%.Also: the batch-size ceiling no longer sticks after a transient CUDA error; the recents feed (#721), an opt-in combined feed at
/recents; publish dates stored as true UTC; chapters listed in episode descriptions (#720); an episode Download menu.2.96.2: Whisper pool and rate-limit hold
The hold parked each 429'd episode as
deferredand leaned on a maintenance pass to return it. The gate was also skipped by user requests and by API-started runs. A 429 now returns the episode to pending in place, keeping its priority, and the gate covers every entry point.The Whisper pool is opt-in and off by default. Against a remote backend that accepts parallel requests, it runs several episodes at once under a single cap on in-flight requests. Both dials are settings.
2.96.3: stats counted reprocesses twice
The dashboard and per-podcast stats summed a join, so a reprocessed episode's time saved counted once per run (#727). Length, time saved, and episode count now come from the episode's current row. Per-run costs still count every run. The lifetime counter credits only the change since it was last credited, with a backfill.
2.96.4: probes
A rate-limit hold re-probes instead of waiting out a reset that may no longer be true, through
llmUsageUrlor a single minimal completion.A Whisper backend's optional health endpoint is sampled for replica count, model, and total concurrency, with a suggested cap. It is bounded: only while the pool is active, cached 120 seconds, a 15 second sampling budget, one probe per backend. When sampling cannot prove it wrapped the replica set, the UI says "at least N".
Version
2.96.5
Test plan
pytest tests/(6827 passed), ruff cleantsc --noEmit,npm run lint, production build,vitest(832 passed)/api/v1/healthreports 2.96.4; the health probe resolves a three-replica backend with no mismatch