Repository navigation
Analytics window, and the demo data that made it possible to build - #3
Merged
Merged
Conversation
…uild Analytics are impossible to build against an empty database and misleading to build against a handful of test rows. Five submissions make every chart look fine: no long tail, no skew, no segment small enough to be noisy, no date range wide enough to need a scale. The first honest look at a report is the moment somebody points it at a real form with four hundred responses, which is much too late to find out the bar chart cannot cope with a label longer than twelve characters. So this is two things, and the order matters: a generator that produces a survey and five hundred people who answered it, and a report built against that rather than against wishful thinking. ## The demo data A team pulse survey — seven questions chosen for what they make possible downstream rather than for what a pulse survey would ideally ask. Two categorical dimensions to group by, a 0–10 recommendation score, a 1–5 rating, a lumpy small integer, a multi-select whose percentages deliberately sum past 100, and an optional free-text box answered by well under half. Every submission goes through `atf_process_submission()` — the same function a stranger's POST reaches. Sanitising, validation, spam screening, storage, the counters and `atf_entry_created` all run exactly as in production, and the request carries a properly signed timestamp rather than being waved past the time trap. A seeder that wrote rows directly would produce data the pipeline could never have produced, and the first thing that would hide is a bug in the pipeline. The population is not random, because random answers produce flat charts — every question at 1/n, every segment identical, every correlation zero, which looks like a working report and proves nothing. Each respondent gets a latent morale that depends on their team and tenure, and their answers follow from it. Support really is unhappier than Design (NPS -51 against +52), satisfaction really does track the recommendation score, and people really do write comments at the extremes and not in the middle. It is reproducible from a seed, and it comes back out: the form and every entry are tagged, and removal deletes exactly what carries the tag. Never "everything on the demo form" — that form has a working shortcode, so somebody will eventually answer it for real, and there is a test that says their answer survives. ## Developer mode All of it is gated on OpenStation's own per-user Developer mode switch, read rather than duplicated. It writes hundreds of entries into whatever database this happens to be, which is fine on a laptop and not something to leave one hover away from "Themes" on a site collecting real enquiries. The preference is not the permission. It answers "show me these"; the capability answers "you may use them"; both are checked on every route. Treating a user-meta preference as authorisation would mean anybody who can write their own meta could seed a database. ## The report Headline counts, submissions per day, a tally or a distribution per question, NPS for any 0–10 question, and a cross-tab of every numeric answer by any categorical one — which is the difference between "the average is 7.5" and "Support is at 5.8 and everyone else is above 7". Three things that look fine when wrong, so each has a test: - A timeline includes its empty days. Keep only the days that had a submission and the gaps close up, so one spike and three weeks of silence renders as a steady trickle. - NPS is not an average. Promoters minus detractors, as percentages, with the passives counting for nothing. Six sevens and eights average to a healthy-looking 7.5 and are an NPS of exactly zero. - A mean needs its distribution. Everybody answering 3, and half answering 1 with half answering 5, are opposite findings with the same mean. No charting library. Every chart is ordinary elements sized in percentages — a bar chart *is* a list of labelled quantities, and as a list it can be read aloud, selected and searched, which a painted canvas cannot without being described all over again. ## Found while building it - Fifty submissions in one request exhausted a 128MB limit, failing as a bare 500 with the entries already written. WordPress adds every post it touches to the object cache and nothing evicts it, and under WP_DEBUG `$wpdb` keeps every query with a backtrace. Both are now freed per chunk, and a failed chunk is retried rather than abandoning the batch. - The window's own token block listed three roots and not the new fourth, so every `var( --atfb-* )` in the report was invalid. That does not fail loudly: the declaration is simply dropped, and a chart of coloured bars renders as a blank rectangle. - The survey put `notifications` under `settings`, where nothing reads it. Harmless by luck — the default is empty — but the test asserting "no notifications" was passing against a key nothing uses, and would have gone on passing after somebody added one in the right place. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0119U4sRRWGcTQdwYreTwpAp
CI caught what the local run did not: `src/dock.ts` had no exports, so TypeScript did not consider it a module and the test importing it for its side effects failed to typecheck. Vitest never noticed — it transpiles without checking types — so the test passed locally while the build broke. Exporting `submenuFor()` fixes the error and makes the test better than it was. The rows are now asserted by calling a pure function of the config rather than by resetting the module registry and watching a fake shell receive a registration, which is both less machinery and a test somebody might actually extend. Also swaps `Array.at()` for index arithmetic: the project targets ES2020, where it does not exist. Same category of miss — the transpiler was happy and the type checker was not. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0119U4sRRWGcTQdwYreTwpAp
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two things, and the order matters.
Why the demo data comes first
Analytics are impossible to build against an empty database and misleading to build against a handful of test rows. Five submissions make every chart look fine — no long tail, no skew, no segment small enough to be noisy, no date range wide enough to need a scale. The first honest look at a report is the moment somebody points it at a real form with four hundred responses, which is much too late to discover the bar chart cannot cope with a label longer than twelve characters.
The survey
Seven questions, chosen for what they make possible downstream rather than for what a pulse survey would ideally ask: two categorical dimensions to group by, a 0–10 recommendation score, a 1–5 rating, a lumpy small integer, a multi-select whose percentages deliberately sum past 100, and an optional free-text box answered by well under half.
These are real submissions. Every one goes through
atf_process_submission()— the same function a stranger's POST reaches. Sanitising, validation, spam screening, storage, the counters andatf_entry_createdall run exactly as in production, and the request carries a properly signed timestamp rather than being waved past the time trap. A seeder that wrote rows directly would produce data the pipeline could never have produced, and the first thing that would hide is a bug in the pipeline.The population is not random, because random answers produce flat charts — every question at 1/n, every segment identical, every correlation zero, which looks like a working report and proves nothing. Each respondent gets a latent morale depending on their team and tenure, and their answers follow from it:
It comes back out. The form and every entry are tagged; removal deletes exactly what carries the tag, never "everything on the demo form" — that form has a working shortcode, so somebody will eventually answer it for real, and there is a test asserting their answer survives.
Developer mode
Gated on OpenStation's own per-user switch, read rather than duplicated. It writes hundreds of entries into whatever database this happens to be.
The preference is not the permission. It answers "show me these";
atf_edit_formsanswers "you may use them"; both are checked on every route. Verified with dev mode off:GETandPOST /demoboth 404, the panel is gone, the dock row is gone, and the report itself still renders.The report
Headline counts, submissions per day, a tally or distribution per question, NPS for any 0–10 question, and a cross-tab of every numeric answer by any categorical one — the difference between "the average is 7.5" and "Support is at 5.8 and everyone else is above 7".
Three things that look fine when wrong, so each has a test:
No charting library. Every chart is ordinary elements sized in percentages — a bar chart is a list of labelled quantities, and as a list it can be read aloud, selected and searched.
Bugs found while building it
WP_DEBUG,$wpdbkeeps every query with a backtrace. Both freed per chunk, chunk size lowered, and a failed chunk now retries instead of abandoning the batch.var( --atfb-* )in the report was invalid — which does not fail loudly: the declaration is dropped and a chart of coloured bars renders as a blank rectangle.notificationsundersettings, where nothing reads it. Harmless by luck, but the test asserting "no notifications" was passing against a key nothing uses and would have gone on passing after somebody added one in the right place.Verified
516 PHPUnit (36 new), 202 Vitest (8 new), PHPCS clean. The full 500-submission run was driven from the browser end to end; the dev-mode gate was checked from both sides; the demo-data row's gate is mutation-tested.
🤖 Generated with Claude Code
https://claude.ai/code/session_0119U4sRRWGcTQdwYreTwpAp