feat(dashboard): show metrics at cluster, namespace and shared scope by Hanson-Tsai · Pull Request #744 · otterscale/dashboard

Hanson-Tsai · 2026-10-14T13:21:49Z

Summary

Jira: OTTERSCALE-288. Until now every dashboard page asked the hub for cluster-wide PromQL, and the hub answered anyone who was signed in. A Workspace member could read every namespace's metrics. The hub now answers at three scopes (otterscale/otterscale#310), and this PR makes the dashboard ask at the right one.

scope	hub route	who	used by
cluster	`/prometheus/...`	cluster-admin, or anyone bound to `otterscale:metrics-cluster-view`	Cluster and Storage dashboards; every page for those users
namespace	`/prometheus-tenancy/...?namespace=<ns>`	anyone who may `get pods` in that namespace	Overview, Model and Compute pages for everyone else; the hub pins every selector to the namespace
shared	`/prometheus-shared/<name>`	any Workspace member	GPU, KubeVirt and Ceph totals: per card, per node or platform-wide, never per Workspace or Pod

What changes

Access check (src/lib/server/metrics-access.ts): the layout asks the hub once whether the user has cluster metrics access, with three results: allowed, denied, unknown. A failure is retried once. unknown renders as no access, with a banner, and does not redirect.
Clients (src/lib/prometheus.ts): workspacePrometheusDriver picks cluster or namespace scope; sharedMetrics calls the named shared queries. sharedMetrics always sends the current Workspace's namespace when there is one. It doesn't use the namespace picked in the admin's selector.
Pages:
- Overview, Model and Compute stay in the user's namespace.
- Cluster and Storage are hidden from the sidebar without access. Opening them by URL lands on the overview with a notice.
- The storage capacity notice, the two KubeVirt cards, GPU memory and XID, the Update dialog's GPU Resources, and a model service's GPU Allocation now come from the shared scope. Members see them too.
- A node's GPU Allocation stays cluster-only: it lists every namespace's Pods on the node.
Failures read as unknown, never as zero:
- XID and the Pod & hardware health cells show a grey "—".
- GPU memory and the KubeVirt cards show "No data to display".
- GPU Resources shows an error, not "0 devices".
- The GPU Allocation diagram still draws the placement, without usage.
Guard test (namespace-scope.spec.ts): scans every directory Workspace pages use. It fails on infrastructure PromQL, cluster-scope drivers, new PrometheusDriver and the infrastructure helpers, and requires the shared-scope components to actually call the shared client.
docs/metrics.md: the rules for developers.

Review fixes in this update

GPU Allocation lists a node's Pods itself only for a cluster-admin. Everyone else, including users bound to metrics-cluster-view, gets the shared per-card totals, so the cards no longer undercount silently.
The Workspace picker on Compute and Model shows only when the user is a cluster-admin and cluster access is allowed.
GPU utilisation lookups are batched by encoded URL length (the hub caps the forwarded URI at 6144 bytes), still at most 256 cards per request.
The failure states above; every failure is logged with console.warn.
The e2e also calls the hub directly: 403 for cluster scope, for another namespace, and for shared without a namespace. Waits use waitForResponse / expect.poll instead of fixed timeouts.

Screenshots

Taken on a single-node lab cluster against this branch (4c157c2) and hub b229eb5. Left: a cluster-admin. Right: a view member of one Workspace. Under each pair are the metrics requests that step sent. The signed-in user's name in the sidebar footer is covered.

Overview

Cluster and Storage: the member lands on the overview

Compute: KubeVirt cards from the shared scope

Model analytics: GPU memory and XID from the shared scope

Model service: GPU Allocation

Model service: GPU Resources in the Update dialog

Node menu: GPU Allocation stays cluster-only

Create VM (control: unchanged for both)

For the member, no step sent a cluster-scope request, every shared request returned 200 with the Workspace's namespace, and no page component sent infrastructure PromQL. The images live on the pr-assets/otterscale-288 branch, which is not meant to be merged.

Compatibility

This dashboard needs the hub from otterscale/otterscale#310, and the two must be upgraded together:

New hub, old dashboard: regular users get 403 on every chart.
Old hub, new dashboard: the namespace and shared scopes return 404.

Compute charts also need the kubevirt module at chart 1.2.0 (otterscale/helm-charts#67), which keeps a VM metric's namespace as the VM's own.

Testing

pnpm check: 0 errors, 0 warnings. pnpm lint: clean.
pnpm test:unit: 13 files, 90 tests.
e2e/tenant-metrics.test.ts: 9/9, run on the lab cluster as a regular user against this branch.
The screenshot comparison above: 20 steps, identical to the previous accepted run.

Regular users query the Workspace's namespace through the tenancy proxy. VM metrics are selected on namespace rather than exported_namespace: the kubevirt PodMonitor now keeps the VMI's own namespace label, which is the one namespace scope enforces. The KubeVirt controller and worker tiles are cluster scope and are dropped for viewers without cluster metrics access, so their queries are never sent. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Regular users query the Workspace's namespace through the tenancy proxy. GPU memory and XID come from DCGM, which carries gpu-operator's namespace, so viewers without cluster metrics access see neither and send neither query; the vLLM, kube-state-metrics and cAdvisor panels are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The GPU allocation dialog and the GPU resources modal read nodes, DCGM and HAMi at cluster scope, where a regular user only gets 403. They now render, and query, only for viewers with cluster metrics access, and the actions menus drop the GPU allocation row with them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Against a deployed dashboard: no cluster or storage dashboards, every chart query goes to /prometheus-tenancy for their namespace only, the overview has data, and no infrastructure metric is asked for. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…returns The lab serves Keycloak on the dashboard's origin under /auth/, so an origin check alone resolved while still on the password page and the next navigation could abort the login callback. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…sDriver The cluster and storage dashboards and the workspaces viewer still built their PrometheusDriver by hand, against the rule docs/metrics.md states. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

A transient hub error silently hid Cluster and Storage from an admin. A non-ok answer now logs its status, a thrown error its cause, with the cluster; both still fail closed. A denial is not logged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The absence checks could pass before the sidebar had rendered at all. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

It sat inside the private-CA setup steps. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…ible On the overview the Dashboard group is collapsed, so its sub-items are in the DOM but hidden. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

… no ResourceQuota Prometheus keeps no metadata for a kube-state-metrics family without series, so the "is the resourcequota collector on" check failed on any cluster without a single ResourceQuota and every workspace showed an error. Decide from kube_namespace_created and kube_resourcequota alone. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

- ask a failed access check once more before giving up - show a banner on unknown instead of quietly drawing the regular view - redirect only on a denial, so an admin keeps their bookmark

- wait one tick before replaceState on a first page load - keep the toast where it was

- add dashboardBody to pick charts, loading, no-namespace or empty - show a namespace-unknown notice instead of an endless spinner - stop the compute dashboard going blank without a driver

- delete resource-viewer/viewers/workspaces-viewer.svelte (unused) - raise the tenant-metrics describe timeout to 120 s for Keycloak login - give the login waitForURL a 60 s timeout

- namespace="" matched only series without the label - so an admin choosing all namespaces saw nothing

- the server rewrites selectors the way prom-label-proxy does it - docs and the prometheus.ts header no longer say it is forced - metrics.md points optional namespace filters at namespaceSelector

- cluster and storage are hidden from the menu; a bookmark still opens - the retry spec also covers a thrown error followed by an answer

- sends a query name and parameters, never promql - namespace is the workspace's, left out when it has none - returns prometheus-query results, so callers keep their types

- the kubevirt component tiles ask the shared scope for every viewer - so does the overview's storage card for the cluster's capacity

- per-card inventory and vgpu totals come from the shared scope

- the model's nodes come from its namespace, as before - their gpu numbers come from the shared scope, for every viewer - xid shows as unknown, not 0, when it cannot be asked - drop vllmModelHostnamesSelector, which nothing uses any more

- utilization comes from the shared scope for every viewer - members see their own pods and the scheduler's per-card totals - the node view still lists every pod, so it stays with admins

- gpu, kubevirt and ceph components may only use the shared scope - e2e follows every shared call: known name, answered, own namespace - e2e opens the model's analytics, gpu allocation and gpu resources

- fetchModelNodes resolves null when the lookup fails, [] for no nodes - gpu memory shows no data, not no gpu, when a lookup or call fails - xid shows the grey unavailable dash when the node lookup fails - test fetchModelNodes for a failed lookup and duplicate nodes

- metrics.md and the prometheus.ts header still named two scopes

- the row behind the dialog has gpu icons too - the dialog's overlay took the click meant for the row's icon - found when the e2e first ran on the hci lab with E2E_MODEL set

- list node pods only for a cluster admin, who may list every namespace - fetch gpu-allocated totals for everyone else, metrics viewers included - align card-usage and fetch-topology comments with the new check

- render namespace picker when admin and cluster metrics both allowed - keep an admin with unknown access on their own workspace

- size batches so gpu-fb-total stays under the hub's 6144-byte uri cap - count each uuid as the hub escapes and url-encodes it, five matchers - keep the 256-values cap for short uuids - test long, typical and short uuids against a replica of the hub query

- the podmonitor label change ships in kubevirt 1.2.0, not 1.1.2

- scan workspace, model, compute, gpu-allocation, kind-viewer and routes - leave out the cluster and storage dashboards, admins only - forbid new PrometheusDriver and infra helpers from $lib/prometheus - check the listed helpers are still exported, so none goes stale - require actual shared-scope calls instead of the word shared

- add fetchCombinedInstantOrNull, null on failure, with a warning - grey out restart, oom, throttle, shm and preempt when it fails - warn when the gpu memory card fails to load - clear kubevirt cards on a failed reload and show no data - show an error in gpu resources instead of 0 devices - add pod_health_unavailable and gpu_resources_fetch_error messages

- carry the workspace's namespace when there is one, omit it otherwise

- ask cluster scope, another namespace and shared without one directly - expect 403 for each, and 200 for shared as the user's own workspace - wait for the shared answers each step sends, registered before it - poll for chart data instead of a fixed three-second sleep

- the chained replace read as double escaping to code scanning - same byte counts; tests pin backslash, quote and metacharacters

- source-map-js 1.2.2 and postcss-selector-parser 7.1.6 - static-copy watches with chokidar 4, which drops unpatched braces - pnpm audit: no known vulnerabilities

- bring the branch up to date with main; no conflicts

Hanson-Tsai and others added 30 commits October 8, 2026 19:18

docs(dashboard): which metrics a namespace-scoped page may use

f9ac60e

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

refactor(dashboard): every cluster-scope driver from clusterPrometheu…

44c5fa6

…sDriver The cluster and storage dashboards and the workspaces viewer still built their PrometheusDriver by hand, against the rule docs/metrics.md states. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

test(dashboard): see the sidebar before checking links are absent

8f8b79b

The absence checks could pass before the sidebar had rendered at all. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

docs(dashboard): metrics doc link under Documentation

bc016f5

It sat inside the private-CA setup steps. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

test(dashboard): sidebar e2e expects the model link attached, not vis…

9b1b9e8

…ible On the overview the Dashboard group is collapsed, so its sub-items are in the DOM but hidden. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

test(dashboard): e2e comment no longer mentions metadata

21dc7f8

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

feat(dashboard): tell unknown cluster metrics access apart from a denial

6f2c8b6

- ask a failed access check once more before giving up - show a banner on unknown instead of quietly drawing the regular view - redirect only on a denial, so an admin keeps their bookmark

feat(dashboard): say why a cluster dashboard sent you to the overview

9db937b

fix(dashboard): strip the notice only after the router has started

1b138b8

- wait one tick before replaceState on a first page load - keep the toast where it was

fix(dashboard): explain an unknown workspace namespace, don't spin

d25c61b

- add dashboardBody to pick charts, loading, no-namespace or empty - show a namespace-unknown notice instead of an endless spinner - stop the compute dashboard going blank without a driver

chore(dashboard): drop the unused workspaces viewer, give e2e login room

3eb5b37

- delete resource-viewer/viewers/workspaces-viewer.svelte (unused) - raise the tenant-metrics describe timeout to 120 s for Keycloak login - give the login waitForURL a 60 s timeout

fix(dashboard): compute's all-namespaces choice selects every namespace

fe4a39e

- namespace="" matched only series without the label - so an admin choosing all namespaces saw nothing

docs(dashboard): say the namespace rewrite is prom-label-proxy's

6af47e9

- the server rewrites selectors the way prom-label-proxy does it - docs and the prometheus.ts header no longer say it is forced - metrics.md points optional namespace filters at namespaceSelector

fix(dashboard): say the unknown-access banner hides the menu entries

8c6b467

- cluster and storage are hidden from the menu; a bookmark still opens - the retry spec also covers a thrown error followed by an answer

feat(dashboard): add a client for the server's named shared queries

cded6ec

- sends a query name and parameters, never promql - namespace is the workspace's, left out when it has none - returns prometheus-query results, so callers keep their types

feat(dashboard): show kubevirt and storage capacity cards to members

88d4993

- the kubevirt component tiles ask the shared scope for every viewer - so does the overview's storage card for the cluster's capacity

feat(dashboard): show gpu resources in the update dialog to members

372ac47

- per-card inventory and vgpu totals come from the shared scope

feat(dashboard): show a model's gpu allocation to members

0094ae2

- utilization comes from the shared scope for every viewer - members see their own pods and the scheduler's per-card totals - the node view still lists every pod, so it stays with admins

test(dashboard): check members get shared numbers and nothing more

5f9bf72

- gpu, kubevirt and ceph components may only use the shared scope - e2e follows every shared call: known name, answered, own namespace - e2e opens the model's analytics, gpu allocation and gpu resources

docs(dashboard): say the server answers at three scopes

b4a4778

- metrics.md and the prometheus.ts header still named two scopes

Hanson-Tsai added 9 commits October 14, 2026 16:51

test(dashboard): look for the gpu button inside the update dialog

c08c688

- the row behind the dialog has gpu icons too - the dialog's overlay took the click meant for the row's icon - found when the e2e first ran on the hci lab with E2E_MODEL set

fix(dashboard): use shared gpu totals unless cluster admin

65d8214

- list node pods only for a cluster admin, who may list every namespace - fetch gpu-allocated totals for everyone else, metrics viewers included - align card-usage and fetch-topology comments with the new check

fix(dashboard): show workspace picker only at cluster scope

f8f1a4e

- render namespace picker when admin and cluster metrics both allowed - keep an admin with unknown access on their own workspace

docs(dashboard): name kubevirt chart 1.2.0 in the guard test

f18754a

- the podmonitor label change ships in kubevirt 1.2.0, not 1.1.2

docs(dashboard): say shared requests may omit the namespace

9184549

- carry the workspace's namespace when there is one, omit it otherwise

This was referenced Oct 14, 2026

feat: guard metrics in the cluster and authorize them per scope at the hub otterscale/otterscale#310

Open

feat(prometheus-stack): guard Prometheus, Alertmanager and exporters behind authentication otterscale/helm-charts#67

Open

github-advanced-security Bot found potential problems Oct 14, 2026

View reviewed changes

Comment thread src/lib/components/gpu-allocation/fetch-utilization.ts Fixed

Hanson-Tsai added 3 commits October 15, 2026 10:29

fix(dashboard): count escaped uuid length per character

7dfc0d4

- the chained replace read as double escaping to code scanning - same byte counts; tests pin backslash, quote and metacharacters

fix(deps): override vulnerable build-time packages

32a10ac

- source-map-js 1.2.2 and postcss-selector-parser 7.1.6 - static-copy watches with chokidar 4, which drops unpatched braces - pnpm audit: no known vulnerabilities

chore: merge main into the otterscale-288 branch

9e64814

- bring the branch up to date with main; no conflicts

Hanson-Tsai marked this pull request as ready for review October 15, 2026 05:53

Hanson-Tsai requested review from KUASWoodyLIN, ZhangEnYao and iasthc as code owners October 15, 2026 05:53

Navigation Menu

Repository navigation

feat(dashboard): show metrics at cluster, namespace and shared scope - #744

feat(dashboard): show metrics at cluster, namespace and shared scope#744
Hanson-Tsai wants to merge 42 commits into
mainfrom
feat/OTTERSCALE-288-kube-rbac-proxy

Hanson-Tsai commented Oct 14, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

Hanson-Tsai commented Oct 14, 2026

Summary

What changes

Review fixes in this update

Screenshots

Compatibility

Testing

Related

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants