Render every dashboard chart from API data instead of PNG images #489

🧑‍🔬 Advanced Issue — the most complex work in this project: architectural, multi-module, or core-logic changes where the solution itself may need discovering.
Time: ~30+ hours · Prerequisites: a proven track record here (≥1 completed intermediate issue; demonstrated CI/CD proficiency substitutes for workflow-focused issues).
The bar is production-ready: safe, maintainable, architecturally sound.

The task

Extends #333 (assigned to @phillip-nyinomujuni): #333 migrates the first chart, the contributor heatmap; this issue covers the whole catalogue. The design discussion there applies here. In particular, @phillip-nyinomujuni's finding that CoverageMatrix.tsx reads only the ramp's length and takes its colours from the --heat-* CSS tokens, and @exploreriii's advice to normalise unbounded activity counts when the data is exported rather than fixing bucket constants.

Problem: Every chart on the dashboard is a matplotlib PNG shown as an image. ChartSectionCard.tsx renders <img src={chartUrl(active.file)}> for every variant (web/src/components/ChartSectionCard.tsx:57-58 on main), and a chart variant in the API carries only an image filename and its pixel size (ChartVariant, web/src/api.ts:44). As a result:

• No exact values: a reader cannot hover a bar or read a number, and cannot see the data behind a chart without downloading a CSV.
• No theming or crisp text: images do not follow dark mode and render at a fixed DPI, and their labels are rasterised text.
• Pre-rendered variants: each time window (week, month, year, all time) is a separate PNG, so a reader cannot change mark, scale or span.
• Limited accessibility: alt text is all an assistive-technology user gets.
• No exploration: nothing links a chart to the tables below it.

Across the two published organisations there are 62 chart variants (43 for hiero-ledger, 19 for hiero-hackers). There is no chart-data contract in the API, so no chart can move without designing one.

Impact / what done looks like:

• A typed chart-data contract in API v1, added without breaking v1.
  • Each chart variant gains an optional interactive reference to a JSON document under data/api/v1/<org>/charts/. PNG generation continues for external reports, and an older bundle without chart data still loads.
  • Documents come in a small set of kinds, one TypeScript interface each: timeseries, categories, matrix, network, events. They share metadata: unit, population (the counting rule), window, dimensions and freshness.
  • Drawing options are declared per chart in the dashboard_spec/ families, not coded per chart. Examples: mark, stacking, normalisation, ranking, top N, a reference line, and a value ceiling.
• Every declared chart variant has a document.
  • The exporter builds each document from a CSV the owning pipeline already writes, and never aggregates, so the numbers stay the analysis's own.
  • A spec test requires every declared chart to name a data source.
  • The output contract runs the pipelines on fixture data and requires every produced variant to have a loadable document. That is a deliberate contract update, not a test to work around.
• The React app renders only interactive charts. When a document is missing it says so plainly, and a failed request offers Retry. It never falls back to an image. Each chart provides:
  • exact values on hover and keyboard focus;
  • a Chart / Data switch;
  • a CSV of the selected rows;
  • an expanded view;
  • light and dark themes from the existing tokens;
  • the chart's counting rule and freshness in its footer.
• Calendar and counting rules hold.
  • Flows fill empty buckets with zero; stocks carry their level forward.
  • Point-in-time snapshots are never gap-filled.
  • Partial buckets are flagged.
  • Distinct-people counts are never summed across periods: span controls select existing buckets and never re-aggregate them.
• Tests: uv run pytest, uv run ruff check src tests, and in web/ npm test, npm run lint, npm run build all pass. The branch implementing this is feat/web-shadcn-ui; its PR will link this issue.

Known unknowns and risks:

• Per-kind documents or a generic chart grammar. The implementation takes the per-kind route that Render the first chart client-side from API data (contributor heatmap) #333 recommends: five interfaces, with drawing declared in specs. A generic, vega-lite-style schema was rejected as unnecessary for this catalogue. Reviewers should push back if a chart does not fit a kind.
• Parity with the PNGs has to be checked, not eyeballed. Values, ordering, bucket edges and colour ramps must match. The one deliberate divergence is the Scorecard breakdown, now a repository × check matrix on a fixed 0–10 scale. Stacked bars implied the checks add up to the score, but Scorecard weights them.
• Network layout. The PNG's spring layout packs the dense core on top of itself. The browser starts from the exported positions and relaxes them deterministically, with a collision radius that includes each label, so every reader sees the same picture. A repository × repository matrix view is offered for dense graphs.
• Manual inputs. The Discord charts come from a manual archive, not a scheduled fetch. Their data files keep the archive date, so a later export must not extend the series to today.
• Bundle size. Recharts ships in a lazily loaded chunk of about 126 kB gzipped, and pages without charts do not load it.
• Overlapping in-flight work (rebase and review order to be agreed with the authors):
  • PR fix: improve chart and tab accessibility #422 (Make chart alt text variant-aware and expose active tabs to assistive tech #326): it edits image alt text in ChartSectionCard.tsx and TabBar.tsx. This work removes the images, and TabBar.tsx no longer exists, so Make chart alt text variant-aware and expose active tabs to assistive tech #326's image half becomes moot. Its tab-state half is already covered: variant tabs are radio groups that report the checked tab, and the sidebar tabs carry aria-current="page".
  • PR feat(web): add printable dashboard tabs with provenance #473 (Render any tab as a clean printed document for TSC meetings #356, printable tabs): it prints chart PNGs. Printed charts need to come from the interactive render instead.
  • PR Feat/playwright smoke 332 #391 (Add a Playwright smoke suite for the built dashboard #332, Playwright smoke): some of its assertions target chart images and will need to target the rendered charts.

How to work on this

1. Claim it: comment /assign — yes, even at this level; unassigned PRs are closed automatically.
2. Propose your design as a comment before building. Cover the approach, the alternatives you rejected, and the system-wide impact. For large changes, say how you'll split the work into reviewable PRs. (@coderabbitai plan can sketch a starting point, but the design is yours.)

What tends to bite experienced contributors in this repo:

• The dependency DAG in docs/architecture.md is deliberately strict (zero violations today) — a solution that needs a new cross-layer import needs a design conversation first, not an exception.
• The output-contract test pins every CSV, chart, and dashboard artifact the pipelines produce. Changing the output surface means changing the contract on purpose, and downstream dashboard consumers exist.
• Ingestion has deliberate two-layer staleness semantics (TTL cache vs. the durable incremental dataset store with reuse/refresh windows) — read the module docstrings before touching fetch paths; naive "fixes" here reintroduce bugs we've already removed.

🤖 AI: tools are welcome; verified work is required — see the AI policy. Fully automated bot PRs are closed.

Before opening your PR:

• The PR includes a short design/impact note: approach, alternatives considered, affected modules, compatibility impact
• Correctness, safety, and performance are evaluated, not assumed — and the evaluation is visible in the note or the tests
• Testing is comprehensive, including deliberate output-contract updates where the output surface changes
• I reviewed my own diff as if it were someone else's

Review expectations: advanced PRs get probing questions and may take longer than a day — that's the level working as intended. Stuck or want a design sounding board? Comment here — see getting help.