Skip to content

Analytics window, and the demo data that made it possible to build - #3

Merged
AllTerrainDeveloper merged 2 commits into
mainfrom
demo-data-and-analytics
Aug 16, 2026
Merged

AllTerrainDeveloper merged 2 commits into
mainfrom
demo-data-and-analytics

Conversation

@AllTerrainDeveloper

Copy link
Copy Markdown
Owner

Two things, and the order matters.

Why the demo data comes first

Analytics are impossible to build against an empty database and misleading to build against a handful of test rows. Five submissions make every chart look fine — no long tail, no skew, no segment small enough to be noisy, no date range wide enough to need a scale. The first honest look at a report is the moment somebody points it at a real form with four hundred responses, which is much too late to discover the bar chart cannot cope with a label longer than twelve characters.

The survey

Seven questions, chosen for what they make possible downstream rather than for what a pulse survey would ideally ask: two categorical dimensions to group by, a 0–10 recommendation score, a 1–5 rating, a lumpy small integer, a multi-select whose percentages deliberately sum past 100, and an optional free-text box answered by well under half.

These are real submissions. Every one goes through atf_process_submission() — the same function a stranger's POST reaches. Sanitising, validation, spam screening, storage, the counters and atf_entry_created all run exactly as in production, and the request carries a properly signed timestamp rather than being waved past the time trap. A seeder that wrote rows directly would produce data the pipeline could never have produced, and the first thing that would hide is a bug in the pipeline.

The population is not random, because random answers produce flat charts — every question at 1/n, every segment identical, every correlation zero, which looks like a working report and proves nothing. Each respondent gets a latent morale depending on their team and tenure, and their answers follow from it:

Team n Office days Recommend NPS Satisfaction
Design 44 2.18 8.61 +52 3.86
Data 51 2.04 8.43 +39 3.80
Engineering 136 1.84 7.76 +14 3.51
Sales 77 2.82 7.19 −14 3.27
Support 63 2.40 5.78 −51 2.56

It comes back out. The form and every entry are tagged; removal deletes exactly what carries the tag, never "everything on the demo form" — that form has a working shortcode, so somebody will eventually answer it for real, and there is a test asserting their answer survives.

Developer mode

Gated on OpenStation's own per-user switch, read rather than duplicated. It writes hundreds of entries into whatever database this happens to be.

The preference is not the permission. It answers "show me these"; atf_edit_forms answers "you may use them"; both are checked on every route. Verified with dev mode off: GET and POST /demo both 404, the panel is gone, the dock row is gone, and the report itself still renders.

The report

Headline counts, submissions per day, a tally or distribution per question, NPS for any 0–10 question, and a cross-tab of every numeric answer by any categorical one — the difference between "the average is 7.5" and "Support is at 5.8 and everyone else is above 7".

Three things that look fine when wrong, so each has a test:

  • A timeline includes its empty days. Keep only the days that had a submission and the gaps close up, so one spike and three weeks of silence renders as a steady trickle.
  • NPS is not an average. Six sevens and eights average to a healthy-looking 7.5 and are an NPS of exactly zero.
  • A mean needs its distribution. Everybody answering 3, and half answering 1 with half answering 5, are opposite findings with the same mean.

No charting library. Every chart is ordinary elements sized in percentages — a bar chart is a list of labelled quantities, and as a list it can be read aloud, selected and searched.

The analytics window

Bugs found while building it

  • Fifty submissions in one request exhausted a 128MB limit, failing as a bare 500 with the entries already written. WordPress adds every post it touches to the object cache and nothing evicts it; under WP_DEBUG, $wpdb keeps every query with a backtrace. Both freed per chunk, chunk size lowered, and a failed chunk now retries instead of abandoning the batch.
  • The window's token block listed three roots, not the new fourth, so every var( --atfb-* ) in the report was invalid — which does not fail loudly: the declaration is dropped and a chart of coloured bars renders as a blank rectangle.
  • The survey put notifications under settings, where nothing reads it. Harmless by luck, but the test asserting "no notifications" was passing against a key nothing uses and would have gone on passing after somebody added one in the right place.

Verified

516 PHPUnit (36 new), 202 Vitest (8 new), PHPCS clean. The full 500-submission run was driven from the browser end to end; the dev-mode gate was checked from both sides; the demo-data row's gate is mutation-tested.

🤖 Generated with Claude Code

https://claude.ai/code/session_0119U4sRRWGcTQdwYreTwpAp

AllTerrainDeveloper and others added 2 commits August 16, 2026 01:44
…uild

Analytics are impossible to build against an empty database and misleading
to build against a handful of test rows. Five submissions make every chart
look fine: no long tail, no skew, no segment small enough to be noisy, no
date range wide enough to need a scale. The first honest look at a report
is the moment somebody points it at a real form with four hundred
responses, which is much too late to find out the bar chart cannot cope
with a label longer than twelve characters.

So this is two things, and the order matters: a generator that produces a
survey and five hundred people who answered it, and a report built against
that rather than against wishful thinking.

## The demo data

A team pulse survey — seven questions chosen for what they make possible
downstream rather than for what a pulse survey would ideally ask. Two
categorical dimensions to group by, a 0–10 recommendation score, a 1–5
rating, a lumpy small integer, a multi-select whose percentages
deliberately sum past 100, and an optional free-text box answered by well
under half.

Every submission goes through `atf_process_submission()` — the same
function a stranger's POST reaches. Sanitising, validation, spam screening,
storage, the counters and `atf_entry_created` all run exactly as in
production, and the request carries a properly signed timestamp rather than
being waved past the time trap. A seeder that wrote rows directly would
produce data the pipeline could never have produced, and the first thing
that would hide is a bug in the pipeline.

The population is not random, because random answers produce flat charts —
every question at 1/n, every segment identical, every correlation zero,
which looks like a working report and proves nothing. Each respondent gets
a latent morale that depends on their team and tenure, and their answers
follow from it. Support really is unhappier than Design (NPS -51 against
+52), satisfaction really does track the recommendation score, and people
really do write comments at the extremes and not in the middle.

It is reproducible from a seed, and it comes back out: the form and every
entry are tagged, and removal deletes exactly what carries the tag. Never
"everything on the demo form" — that form has a working shortcode, so
somebody will eventually answer it for real, and there is a test that says
their answer survives.

## Developer mode

All of it is gated on OpenStation's own per-user Developer mode switch,
read rather than duplicated. It writes hundreds of entries into whatever
database this happens to be, which is fine on a laptop and not something to
leave one hover away from "Themes" on a site collecting real enquiries.

The preference is not the permission. It answers "show me these"; the
capability answers "you may use them"; both are checked on every route.
Treating a user-meta preference as authorisation would mean anybody who can
write their own meta could seed a database.

## The report

Headline counts, submissions per day, a tally or a distribution per
question, NPS for any 0–10 question, and a cross-tab of every numeric
answer by any categorical one — which is the difference between "the
average is 7.5" and "Support is at 5.8 and everyone else is above 7".

Three things that look fine when wrong, so each has a test:

- A timeline includes its empty days. Keep only the days that had a
  submission and the gaps close up, so one spike and three weeks of silence
  renders as a steady trickle.
- NPS is not an average. Promoters minus detractors, as percentages, with
  the passives counting for nothing. Six sevens and eights average to a
  healthy-looking 7.5 and are an NPS of exactly zero.
- A mean needs its distribution. Everybody answering 3, and half answering
  1 with half answering 5, are opposite findings with the same mean.

No charting library. Every chart is ordinary elements sized in percentages
— a bar chart *is* a list of labelled quantities, and as a list it can be
read aloud, selected and searched, which a painted canvas cannot without
being described all over again.

## Found while building it

- Fifty submissions in one request exhausted a 128MB limit, failing as a
  bare 500 with the entries already written. WordPress adds every post it
  touches to the object cache and nothing evicts it, and under WP_DEBUG
  `$wpdb` keeps every query with a backtrace. Both are now freed per chunk,
  and a failed chunk is retried rather than abandoning the batch.
- The window's own token block listed three roots and not the new fourth,
  so every `var( --atfb-* )` in the report was invalid. That does not fail
  loudly: the declaration is simply dropped, and a chart of coloured bars
  renders as a blank rectangle.
- The survey put `notifications` under `settings`, where nothing reads it.
  Harmless by luck — the default is empty — but the test asserting "no
  notifications" was passing against a key nothing uses, and would have
  gone on passing after somebody added one in the right place.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0119U4sRRWGcTQdwYreTwpAp
CI caught what the local run did not: `src/dock.ts` had no exports, so
TypeScript did not consider it a module and the test importing it for its
side effects failed to typecheck. Vitest never noticed — it transpiles
without checking types — so the test passed locally while the build broke.

Exporting `submenuFor()` fixes the error and makes the test better than it
was. The rows are now asserted by calling a pure function of the config
rather than by resetting the module registry and watching a fake shell
receive a registration, which is both less machinery and a test somebody
might actually extend.

Also swaps `Array.at()` for index arithmetic: the project targets ES2020,
where it does not exist. Same category of miss — the transpiler was happy
and the type checker was not.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0119U4sRRWGcTQdwYreTwpAp
@AllTerrainDeveloper
AllTerrainDeveloper merged commit ca37c86 into main Aug 16, 2026
3 checks passed
@AllTerrainDeveloper
AllTerrainDeveloper deleted the demo-data-and-analytics branch August 16, 2026 09:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant