diff --git a/.gitignore b/.gitignore index 8ef30e2c6d..cbdc1e909e 100644 --- a/.gitignore +++ b/.gitignore @@ -107,3 +107,6 @@ tmp/ # Local-only Promptfoo benchmark config (do not commit) evals/promptfooconfig.benchmark.yaml + +# Local input-latency profiling artifacts +devtools/input-latency/results/ diff --git a/devtools/input-latency/README.md b/devtools/input-latency/README.md new file mode 100644 index 0000000000..4a0792ce5a --- /dev/null +++ b/devtools/input-latency/README.md @@ -0,0 +1,93 @@ +# Input Latency Profiling + +Developer-only tooling to measure and compare per-keystroke input latency in SuperDoc's +presentation pipeline. It has two halves: + +- An opt-in `InputLatencyProfiler` inside `@superdoc/super-editor` that emits standard + [User Timing](https://developer.mozilla.org/en-US/docs/Web/API/Performance_API/User_timing) + measures prefixed with `superdoc.input-latency.` while you type. +- A Playwright recorder (`profile.mjs`) that types a fixed sequence into a large local DOCX, + reads those measures back, and writes a reproducible JSON report, plus a comparator + (`compare.mjs`) that diffs a baseline against an "after" report. + +Profiling is **off by default** and **privacy-preserving**: nothing is created unless explicitly +enabled, only timings and document-shape counts are captured, and no document text is ever put in +measure details, reports, or transmitted anywhere. + +## Enabling the profiler + +The profiler is created only when, **before editor construction**, either: + +- `process.env.SD_DEBUG_INPUT_LATENCY` is `1` or `true`, or +- `globalThis.__SD_DEBUG_INPUT_LATENCY__ === true`. + +The recorder sets the global automatically via an init script, so you don't set it by hand when +using `profile.mjs`. When disabled, no profiler, marks, measures, timers, or sample arrays are +allocated; each call site does a single null check. + +Only the initial case is measured: non-composing `beforeinput` events with +`inputType === "insertText"`. + +## Recording a baseline + +1. Start SuperDoc Labs (the dev app) and leave it running on the fixed port: + + ```bash + pnpm dev + ``` + + The recorder expects the app at `http://localhost:9094` by default. + +2. Record a report. The fixture is referenced **by absolute path only** and is never copied into + the repository. The expected fixture is SHA-256 + `f73a0f320aa78c00d13fbadecfd2906a7f12dc95a00b306c4f0a6a18afaad277`, size `280245` bytes. + + ```bash + pnpm profile:input-latency -- \ + --url http://localhost:9094 \ + --fixture /absolute/path/to/large-typing-test.docx \ + --expect-sha256 f73a0f320aa78c00d13fbadecfd2906a7f12dc95a00b306c4f0a6a18afaad277 \ + --label baseline \ + --output devtools/input-latency/results/baseline.json + ``` + + `--expect-sha256` fails fast unless the fixture matches the expected hash. The recorder also + rejects a run that reports console/script errors, an unexpected sample count, an unstable + document shape, or any discarded traces — so an incomplete recording never writes a report. + +3. Compare a later "after" report against the baseline: + + ```bash + pnpm profile:input-latency:compare -- \ + devtools/input-latency/results/baseline.json \ + devtools/input-latency/results/after.json \ + --markdown devtools/input-latency/results/comparison.md + ``` + +## Protocol + +Each report records three fixed **scenarios**: + +| Scenario | Layout | CPU | +| --- | --- | --- | +| `layout-on-cpu-1x` | SuperDoc layout pipeline on | normal | +| `layout-on-cpu-4x` | SuperDoc layout pipeline on | 4× CDP slowdown | +| `layout-off-cpu-1x` | control probe (native ProseMirror only, `?layout=0`) | normal | + +For every scenario the recorder performs **one discarded warm-up run plus five measured runs**. +Each run types **40 lowercase ASCII characters at a 100 ms cadence** into the fixture. The primary +metric is 4×-CPU p95 `inputToNextFrameMs`; raw per-input samples, the five run-level p95 values, and +their median absolute deviation (MAD) are all retained so noise is visible. + +## Materiality + +A p95 delta is considered material only when it exceeds `max(5 ms, 2 × MAD)`, where MAD is the +noisier of the baseline/after run-level p95 MADs. Smaller deltas are within noise and must not be +reported as regressions or improvements. + +## Requirements & hygiene + +- Requires a **headed Chromium** driven by Playwright; it will not run in CI. +- Raw reports live in `devtools/input-latency/results/` and are **gitignored** — they contain + machine-specific timings and are not reproducible across hardware, so they are never committed. + Commit only code, docs, and the comparator, never the measurements. diff --git a/devtools/input-latency/compare.mjs b/devtools/input-latency/compare.mjs new file mode 100644 index 0000000000..845c2cb1e3 --- /dev/null +++ b/devtools/input-latency/compare.mjs @@ -0,0 +1,23 @@ +#!/usr/bin/env node +// Strict comparison CLI: reads two report JSON files, validates they are comparable, prints the +// JSON comparison to stdout, and optionally writes a Markdown summary. Exits non-zero on any +// parse, schema, or fingerprint error. +import fs from 'node:fs/promises'; +import { compareReports, renderMarkdownComparison } from './report.mjs'; + +// `pnpm run