feat: dynamic result classification (backend)#344
Open
okt-konst wants to merge 30 commits into
Open
Conversation
…on models Foundation for dynamic result classification: external bug cache (IssueExt), global cause identity (Issue), per-project triage decision with category and expected flag (IssueRule), and the per-result stamp (ResultClassification).
ClassVar is only meaningful inside a class body; the category->expected default map is a module-level constant.
A failed result stamped by an expected rule on an open issue no longer counts as unexpected; closing the issue or an unexpected rule leaves it counted.
is_result_unexpected no longer references Meta.objects after switching to a meta__type lookup.
…k counts get_run_stats and the dashboard per-run nok aggregate counted every result with an err meta, ignoring classification suppression, so effectively-expected failures (expected rule on an open issue) were still reported as unexpected. Exclude suppressed results via the shared helper; nok uses Exists subqueries to avoid join multiplication from combining two multi-valued relations.
…ion filter filter_by_result_classification annotated _has_err/_suppressed on its returned queryset. The history pipeline later feeds that queryset to HistoryService._finalize_queryset, which re-annotates the same names. Use inline Exists conditions in filter/exclude so the returned queryset carries no annotations, removing fragility and redundant subqueries. Adds a regression test exercising the real chained production path.
… invalidation Adds IssueViewSet and IssueRuleViewSet with close/reopen/deactivate/activate lifecycle actions, admin-gated writes, and run-stats cache invalidation on state changes. filter_backends is disabled on both viewsets because the global auto FilterSet cannot build a filter for IssueRule.parameters (JSONField); queryset narrowing is done explicitly instead.
…gory/issue/untriaged filters
…sify input IssueViewSet/IssueRuleViewSet were ModelViewSets without DEFAULT_PERMISSION_CLASSES, so DRF defaulted to AllowAny. They guarded create/update/custom-actions but inherited DRF's default destroy (DELETE), partial_update (PATCH), and (for IssueRuleViewSet) update (PUT) unguarded, letting unauthenticated clients delete or edit issues/rules and skip cache invalidation. Override these handlers with check_action_permission and invalidate_run_stats. Also validate classify category/scope (reject unknown values with 400), derive active and origin consistently from scope, and use get_object_or_404 with a required project query param for robust 404s. Move mid-file imports to the top.
Add GET /runs/<run_id>/issues/ returning one row per issue with classified results in the run, the distinct result count, the (category, expected) pairs seen, and the optional external bug key. Needed so the run view can show which issues account for its results without the client aggregating raw stamps. Returns a bare list (no pagination) and is ungated, matching the existing issue GETs.
Add GET /api/v2/runs/<run_id>/issues/<issue_id>/results/ returning a bare list of results in a run classified under an issue, each with its run-tree package path, name, obtained result and verdicts. Lets the frontend render the per-issue results as a tree linking back to each result log.
The classify endpoint already accepted a matcher, but _capture_tags always captured only important tags, so a match_all_tags rule would gate on the wrong (too small) tag set and never match runs whose relevant tags differ.
The per-issue rules page needs each rule's test name and, on demand, the latest failures a rule has classified across the project's runs.
Each linear history result now carries its stamped classifications (issue id, title, state, external bug key, category, expected) so the UI can show them in the obtained-result column.
…ppressing expected becomes nullable: True=expected (suppresses), False=unexpected, None=none (marker only). Classify respects an explicit null; suppression math (rule.expected=True) is unchanged.
Add GET /issues/:id/results/ to list every result classified under an issue across all its rules/tests. The History page can't serve this view: it requires a single test name (500s without one) while an issue spans multiple tests. Unions ResultClassification stamps, de-dupes by result, newest-run first. Extract a shared row serializer reused by the per-rule results action.
2 tasks
Apply ruff formatting and resolve ruff check findings on the touched Python files: noqa PLR0913 on the widened history filter helper (matches existing convention), ternary for the classify disposition default, rename pre-existing self-reassignments in TestIterationResultQuerySet, and keep the intentional circular-import-avoiding lazy imports (noqa PLC0415). classification models must import after EventLog (noqa I001).
The [tool.ruff.lint].exclude entry for 'tests' is a no-op (exclude is only honored under [tool.ruff]); with force-exclude the linter was checking explicitly-passed test files. Our PR is the first to add test files that hit this. Tests use Django unittest-style TestCase, so flake8-pytest rules (PT009/ PT027) and B017 don't apply; exclude them at the level that actually works, mirroring how migrations are handled.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Backend for dynamic result classification — marking/classifying failed test results so they're inherited by future runs and can suppress known failures from unexpected counts, coexisting with static TRC.
Core model:
IssueExt(external bug cache),Issue(global cause identity, open/closed lifecycle),IssueRule(per-project: category + tri-stateexpected+ matcher + active flag),ResultClassification(per-result stamp,unique(result, rule)).rule.expected=True AND rule.issue.state='open'. This suppression predicate is centralized in one helper and applied across all run-stats / dashboard / history NOK-derivation seams.API (
interfaces/api_v2/classification.py):IssueViewSet/IssueRuleViewSet/ResultClassifyViewSet/ apply-rules, plus:GET /issues/:id/results/, unions a issue's rules across tests).expected(nullable):nonemarks an issue without changing disposition.History gains
category/issue/untriagedfilters and exposeseffective_expected+issuesper result.Test Plan
python manage.py test bublik.tests.test_classification{,_engine,_api,_readside}— 62 tests green against current upstream main (branch rebased onto latest).Rebased onto current
main(no workflow changes). Frontend lives in a separate PR againstts-factory/bublik-ui.