Unify missing episode definition and refresh AniDB group status periodically - #1421
Open
harshithmohan wants to merge 2 commits into
Open
Unify missing episode definition and refresh AniDB group status periodically#1421harshithmohan wants to merge 2 commits into
harshithmohan wants to merge 2 commits into
Conversation
`AniDB_GroupStatus` was only refreshed by the legacy v1 API, so with a v3-only install the cached release-group progress stopped updating and "released-but-absent" missing-episode counts under-counted. - Enqueue `GetAniDBReleaseGroupStatusJob` for each updated series in `GetUpdatedAniDBAnimeJob`, propagating `ForceRefresh` so scheduled runs keep the `ShouldSkip()` ended-anime gate intact
`/v3/Series` → `Sizes.Missing.Episodes` used an aired-but-absent check while `/v3/MissingEpisodes` and the persisted counts used a released-but-absent check via `AniDB_GroupStatus`, so the same series could report different missing counts. - Added `AnimeEpisodeExtensions.IsMissingEpisode` as the single source of truth: aired, not hidden, no local file, and either no group status data or a group with `CompletionState` complete/finished or `LastEpisodeNumber` past the episode number - Applied the group status gate to `Sizes.Missing.Episodes` in both `GenerateSeriesSizes` and `GenerateGroupSizes` so series and group counts agree; `GenerateSeriesSizes` now requires the per-anime status dictionary so future callers cannot fall back to the divergent aired-but-absent logic - Replaced the non-collecting `GetMissing` raw SQL with cache-based filtering through the shared predicate; the collecting variant stays as raw SQL - Added `AniDB_GroupStatusRepository.GetByAnimeIDs` with batched (1000-per-query) lookups
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Addresses GitHub Discussion #1418 — the definition of "missing episode" was inconsistent across the v3 API, and
AniDB_GroupStatusdata went stale without the legacy v1 API.1. Refresh
AniDB_GroupStatusduring periodic anime updates (e2d0e2d8d)AniDB_GroupStatuswas only refreshed by the legacy v1 API, so on a v3-only install the cached release-group progress stopped updating and "released-but-absent" missing-episode counts under-counted.GetUpdatedAniDBAnimeJobnow enqueuesGetAniDBReleaseGroupStatusJobfor each updated series, propagatingForceRefreshso scheduled runs keep theShouldSkip()ended-anime gate intact (no unnecessary UDP traffic for ended anime).2. Unify the missing-episode definition (
6ec2e3c23)/v3/Series→Sizes.Missing.Episodesused an aired-but-absent check while/v3/MissingEpisodesand the persisted counts used a released-but-absent check viaAniDB_GroupStatus, so the same series could report different missing counts.AnimeEpisodeExtensions.IsMissingEpisodeas the single source of truth: aired, not hidden, no local file, and either no group status data or a group withCompletionStatecomplete/finished orLastEpisodeNumberpast the episode number.Sizes.Missing.Episodesin bothGenerateSeriesSizesandGenerateGroupSizesso series and group counts agree.GenerateSeriesSizesnow requires the per-anime status dictionary so future callers cannot fall back to the divergent aired-but-absent logic.GetMissingraw SQL with cache-based filtering through the shared predicate; the collecting variant stays as raw SQL.AniDB_GroupStatusRepository.GetByAnimeIDswith batched (1000-per-query) lookups.Specials are intentionally unchanged (still bare
HasAired), per the existing comment inAnimeSeriesServiceabout there being no reliable way to check specials.Verification
dotnet build Shoko.Server.sln— 0 errors, 0 warningsdotnet test Shoko.Tests/Shoko.Tests.csproj— 2607 passed, 1 failed (YearlySeasonsTests.RealWorldRegression_LateSeasonPremiere_IsClassifiedOnlyUnderItsOwnSeason, pre-existing on master and unrelated to these changes — fixed separately in test: pin end date in yearly seasons regression test #1420)Both DTO constructors now query the (uncached)
AniDB_GroupStatusrepository:new Series(...)runs oneGetByAnimeIDquery per seriesGenerateGroupSizesruns one batched query per groupOn single-item endpoints (
GET /v3/Series/{id}etc.) this is one extra indexed query against a small table — negligible. But list endpoints (GET /v3/SerieswithpageSize=100,/v3/Group, filter endpoints, dashboard, tree) now issue one extra query per constructed DTO, where previously they made zero group-status queries.Question for maintainers: should this be fixed in this PR (batch-load statuses once per page via
GetByAnimeIDsand thread them through theSeries/Groupconstructors), or left as-is/a follow-up? A batch-loading fix has already been implemented and validated locally, so it can be added quickly if preferred — it's deliberately not part of this PR to keep the diff focused.