Skip to content

Key connection report on Test Group; rename to "Connection Lifecycle" #81

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
marblom007 merged 2 commits into from
Aug 2, 2026
Merged

Key connection report on Test Group; rename to "Connection Lifecycle" #81

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
fe6d475
Key the connection report on Test Group; rename to "Connection Lifecy…
marblom007 Aug 2, 2026
0441dfe
Address review: gate legacy fallback on testGroup absence; dedupe gro…
marblom007 Aug 2, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 20 additions & 10 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,19 +90,29 @@ Allure results, selected by test labels:
| Dashboard | all results (aggregate) |
| Meshery | `project == "Meshery"` |
| Mesheryctl | `project == "mesheryctl"` |
| **Kubernetes Connections** | `epic == "Kubernetes Connections"`; results with no `epic` label also match via the Kubernetes `componentUnderTest` fallback (UI + CLI connection tests, grouped by `client`) |
| **Connection Lifecycle** | `testGroup == "Connection Lifecycle"` (Test Plan Test Group, col B); transitionally, results carrying **no** `testGroup` label also match the legacy `epic == "Kubernetes Connections"` / Kubernetes `componentUnderTest` fallback (UI + CLI connection tests, grouped by `client`) |
| Extension: Remote Provider Layer5 Cloud | `project == "Layer5Cloud"` |
| Extension: Kanvas | `project == "Kanvas"` |

The Kubernetes Connections report is a cross-client behavior lens: connection
tests are tagged at their source (UI Playwright specs, CLI converters) with
`epic="Kubernetes Connections"`, `componentUnderTest`, `testId` (`TC-<n>`), and
`client` (`UI`|`CLI`), sourced from the Meshery Test Plan. Tagged tests still
appear in their `project` report; the Connections report is an additional view.
As a fallback, a result that carries no `epic` label at all is included when its
`componentUnderTest` matches Kubernetes, so results predating the `epic`
convention still appear; a result with a *different* `epic` value is not pulled
in by the component fallback.
The Connection Lifecycle report is a **Test-Group-keyed view**: each test is
tagged at its source (UI Playwright specs, CLI converters) with a `testGroup`
label whose value is the Meshery Test Plan "Latest" tab Test Group (column B).
The report filters on `testGroup == "Connection Lifecycle"`. This pattern
**generalizes to one filtered report per Test Group** - any Test Group can get
its own report by keying a plugin's filter on its `testGroup` value. Tagged
tests still appear in their `project` report; this is an additional
cross-client (`UI`|`CLI`) lens.

Transitionally, a result that carries **no** `testGroup` label at all also
matches the legacy epic-based selector (`epic == "Kubernetes Connections"`, with
a Kubernetes `componentUnderTest` fallback for results carrying no `epic` label)
so the report stays populated with connection results emitted before the
`testGroup` label existed. A result that already carries a `testGroup` is
authoritative - it is never pulled in by the legacy fallback even if its
component looks like Kubernetes - so once every connection result carries
`testGroup`, the fallback can be removed with no change in membership. The
report is published at https://qa.meshery.io/connections/ (the plugin key is
kept stable so the URL is unchanged).

<p style="clear:both;">&nbsp;</p>

Expand Down
67 changes: 53 additions & 14 deletions allurerc.mjs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,20 +13,41 @@ const PROJECTS = {
const isProject = (labels, projectName) =>
labels.find(({ name, value }) => name === "project" && value === projectName);

// Behavior-scoped report keys. A Connections result is tagged at its source
// (UI Playwright specs and CLI converters) with epic="Kubernetes Connections",
// componentUnderTest (Test Plan col C), testId=TC-<n> (Test Plan col A), and
// client (UI|CLI). See https://qa.meshery.io and the meshery test-tagging docs.
// Test-Group-keyed report selector. The general report key is the `testGroup`
// label, whose value is the Meshery Test Plan "Latest" tab Test Group (col B).
// Every Test Group can drive its own filtered report by keying on this label;
// "Connection Lifecycle" is the first consumer. Results are tagged at their
// source (UI Playwright specs and CLI converters) - see the meshery
// test-tagging docs and https://qa.meshery.io.
const isTestGroup = (labels, groupName) =>
labels.some(
({ name, value }) => name === "testGroup" && value === groupName,
);

const hasTestGroup = (labels) => labels.some(({ name }) => name === "testGroup");

// Single source of truth for the Connection Lifecycle Test Group value - used
// for both the report display name and the testGroup filter so the two cannot
// drift apart.
const CONNECTION_LIFECYCLE_GROUP = "Connection Lifecycle";

// --- Transitional epic-based fallback (remove once all connection results
// carry testGroup) -----------------------------------------------------------
// Before this rename the connection report keyed on epic="Kubernetes
// Connections" plus a Kubernetes componentUnderTest fallback. Kept only so the
// report is not empty for results emitted before the testGroup label existed;
// drops once fresh runs of every connection test carry testGroup.
const CONNECTIONS_EPIC = "Kubernetes Connections";

// Matches componentUnderTest values that denote Kubernetes connection behavior.
// Used as a fallback selector when a result predates the epic label.
const CONNECTION_COMPONENT_RE = /kubernetes/i;

// Select a result into the Connections report: prefer the explicit epic label.
// The componentUnderTest fallback applies ONLY to results that carry no epic
// label at all (tagged before the epic convention) - a result with a different
// epic value must not be pulled in just because its component is Kubernetes.
// Select a result into the connection report via the legacy epic label: prefer
// the explicit epic label. The componentUnderTest fallback applies ONLY to
// results that carry no epic label at all (tagged before the epic convention) -
// a result with a different epic value must not be pulled in just because its
// component is Kubernetes.
const isConnectionBehavior = (labels) => {
const hasEpic = labels.some(({ name }) => name === "epic");
if (hasEpic) {
Expand Down Expand Up @@ -123,19 +144,37 @@ export default defineConfig({
groupBy: ["parentSuite", "suite", "subSuite"],
},
},
// Cross-client behavior report: aggregates Kubernetes Connection tests from
// BOTH the UI (project=Meshery) and CLI (project=mesheryctl) pools. It keys
// on the epic label, NOT project, so it is an additional lens - connection
// tests still appear in the Meshery and Mesheryctl reports above.
// Test-Group-keyed report: a filtered view over one shared Allure pool,
// selected by the `testGroup` label (Test Plan col B). This "Connection
// Lifecycle" report aggregates the connection tests from BOTH the UI
// (project=Meshery) and CLI (project=mesheryctl) pools - it keys on
// testGroup, NOT project, so it is an additional lens; those tests still
// appear in the Meshery and Mesheryctl reports above. The same pattern
// generalizes to one filtered report per Test Group: add a plugin whose
// filter is isTestGroup(labels, "<Test Group>").
//
// The plugin key stays `connections` so the published report URL
// (https://qa.meshery.io/connections/) is unchanged; only the display name
// and filter change.
//
// Transitional: the isConnectionBehavior (epic) fallback keeps the report
// populated with connection results emitted before the testGroup label
// existed. It applies ONLY to results carrying no testGroup label at all -
// a result that already carries a testGroup (of any value) is authoritative,
// so a different-group result must not be pulled in via the legacy epic/
// component heuristic. Drop the whole fallback once all connection results
// carry testGroup.
connections: {
import: "@allurereport/plugin-awesome",
options: {
reportName: "Kubernetes Connections",
reportName: CONNECTION_LIFECYCLE_GROUP,
singleFile: false,
reportLanguage: "en",
open: false,
logo: "https://raw.githubusercontent.com/meshery-extensions/qa/refs/heads/master/.github/assets/images/meshery/icon-only/meshery-light-icon.svg",
filter: ({ labels }) => isConnectionBehavior(labels),
filter: ({ labels }) =>
isTestGroup(labels, CONNECTION_LIFECYCLE_GROUP) ||
(!hasTestGroup(labels) && isConnectionBehavior(labels)),
// Group by client (UI vs CLI) first, then the suite hierarchy. The
// awesome plugin (preciseTreeLabels) keeps only label names present on
// at least one result, so results missing "client" fall back to the
Expand Down