diff --git a/CREDITS.md b/CREDITS.md index ee260c9..2b885e5 100644 --- a/CREDITS.md +++ b/CREDITS.md @@ -32,6 +32,8 @@ This is the work that closes the loop: once Patrick's C lands upstream, **DS2 / Patrick has asked to stay off the `ffmpeg-devel` mailing list. The mailing-list submission (cover letter, FATE sample, validation campaign) is being prepared in this repo at [`ffmpeg-upstream/`](ffmpeg-upstream/) and will go out under the submitter's name, on Patrick's behalf, with his explicit consent. See [ffmpeg-upstream/README.md](ffmpeg-upstream/README.md) for the chain of credit going to FFmpeg. +**Also — the DSS SP compact-block fix ([hirparak/dss-codec#14](https://github.com/hirparak/dss-codec/pull/14)).** A third way a DSS SP recording encodes a pause: a short block padded with `0xFF`, which our own decoder read as audio and desynced the rest of the file on. Patrick found it and fixed the demux; we validated it against 90 days of production traffic and adopted it, gated so it touches only the files that were actually broken. The story is [chapter 14](docs/14-the-compact-block-pause.md). + --- ## The supporting cast diff --git a/docs/14-the-compact-block-pause.md b/docs/14-the-compact-block-pause.md new file mode 100644 index 0000000..4cd4d56 --- /dev/null +++ b/docs/14-the-compact-block-pause.md @@ -0,0 +1,127 @@ +# 14 — The compact block: a pause we'd never seen, found by someone else + +> DSS hides a pause three different ways, and this repo now has a chapter for each. Chapter 06: +> the **empty block** (`frame_count = 0`, continuation bytes only). Chapters 07 and 13: the +> **re-sync block** (the next block re-anchors at `2·byte1 − 6`, drop the stale carry). This one +> is the third — the **compact block** — and the only one we didn't find ourselves. It arrived +> as a pull request, from someone already in our credits. + +## The pull request + +Patrick Domack is [in this repo's CREDITS](../CREDITS.md) for the hand-written FFmpeg C port of +the DS2 decoder — the work that, once it lands upstream, makes `.ds2` a first-class FFmpeg format. +So when a PR from him showed up on the Rust crate we build from +([hirparak/dss-codec#14](https://github.com/hirparak/dss-codec/pull/14)), titled *"Fix DSS SP +decode: compact blocks, repack pulse path"*, it had our attention. + +Two chapters ago we shipped a DSS SP decoder and thought the demux was closed. We had never heard +the term **compact block**. That's the uncomfortable, useful thing about a relay of strangers +picking one lock: someone hands you a piece of the mechanism you didn't know was still moving. + +## What a compact block is + +A DSS block is 512 bytes: a 6-byte header, then payload. Normally a block is *full* — its last +frame spills across the boundary into the next block, so the demuxer just concatenates payloads +and lets frames straddle. The header's `byte1` gives the carry offset (`poff = 2·byte1 − 6`) that +keeps the running stream continuous. This is the machinery chapters 07 and 13 are about. + +A **compact block** breaks that pattern. It declares `fc` frames that fit *entirely* inside its +own payload — `fc·42 + poff ≤ 506` — and pads the rest of the 512 bytes with `0xFF`. Nothing +straddles. That padding is not audio; it's a full stop. The recorder wrote it because the dictation +paused and the block closed early. The block that follows starts a fresh segment, so *its* leading +carry offset is spurious and must be skipped. + +Our decoder didn't know any of that. It read the `0xFF` padding as if it were frame bytes, fed it +to the CELP synthesis as a burst of nonsense, and — worse — every frame after the pause was now +mis-aligned, because the byte cursor had swallowed 200 bytes that weren't there. Same audible +symptom as the empty-block bug in chapter 06 (clean, then garble after a pause, then desynced to +the end), a completely different shape on disk. Our re-sync fix from chapter 13 never fired, +because a compact block isn't empty — it has a real `frame_count`. + +Patrick's fix: at a compact block, take only the declared frames, skip the `0xFF` tail, and +restart framing on the next block (skipping its spurious carry). Preserve the swap-parity byte +across the realignment — only true empty blocks reset it, or the first `swap = 1` frame of the +resumed segment loses its high byte and pops. It's the same species of rule as chapters 07 and 13, +for a block type we'd simply never triaged. + +## Does it even matter to us? — a census + +Before touching a decoder that already serves production, we did the thing chapters 03 and 10 +insist on: **measure, don't assume.** We pulled every DSS/DS2 recording that reached the pipeline +over 90 days — **327 files** — and ran each through the production decoder, classified by codec and +scanned for compact blocks. + +- **88.7 %** were **DS2-QP** (16 kHz). Patrick's fix is DSS **SP**; it doesn't touch a single one + of them. +- **8 %** were DSS SP. Within those, files with a genuine *mid-stream* compact pause came to + **about four recordings a quarter**. Real, but the long tail of a long tail. +- The only **hard decode failures in the whole set — four of 327 — were 0-byte or + truncated-at-upload files.** Dead on arrival. No decoder recovers those; they get flagged back to + the sender. + +The census settled the strategy more than the fix did. Our decoder already served ~99 % of real +traffic correctly. The volume lever, if we ever wanted one, is DS2-QP (ten times the files), not +this. And it put a number on something worth saying out loud: **bit-exactness against the Olympus +DLL was never our metric.** A dictation that a human typist can hear, complete and in order, is the +product. A sample that differs by a few LSB is not a defect; a noise burst that desyncs the back +half of a 20-minute *état des lieux* is. Compact blocks caused the second kind. That's why we took +the fix — not for four recordings, but because those four were genuinely broken. + +## Shipping it without touching the 99 % + +The one rule for changing a decoder that runs in production: **the files that work today must keep +producing the same bytes tomorrow.** So the fix went in gated. A file takes the block-aware batch +demuxer *only if* it actually contains a mid-stream compact block — detected conservatively: +declared frames that fit, a tail that is entirely `0xFF`, and a block after it (a *trailing* +compact block is just a short final block, a normal partial end, not a pause). Every other file — +all the QP, every SP recording with an empty-block or re-sync pause — keeps the streaming demuxer +and the chapter-13 re-sync untouched, byte for byte. + +We proved it, not asserted it. On the 26 real DSS SP files from the census: + +- **23 of 26 decoded byte-identical**, old binary vs new. +- The **3 that changed were exactly the mid-stream-compact ones** (one command — five back-to-back + property-inspection dictations from a single bailiff's study, cmd `2946368`, whose blocks carried + **202 and 168 bytes of pure `0xFF`**). Their **duration didn't change** — the fix doesn't add + audio, it re-aligns the audio after the pause, which is the whole point. +- The three other census files the scan flagged as "compact" did **not** move: their compact block + was the *last* one, a normal partial end, and the gate correctly left them alone. +- **25 of 25** sampled DS2-QP files: identical. + +Then it went to the production binary the boring way — backup, atomic replace, a real end-to-end +conversion of a QP file (unchanged) and the five repaired SP files (clean MP3s out). + +## The box we didn't get to open + +Patrick's PR carried a **second** change we did *not* take, and the reason is a chapter-08 story in +miniature. + +Besides the demux fix, the PR reworks the decoder itself — a *repack* pulse-position path he'd made +bit-exact against the Olympus DLL. To decide whether it was worth adopting, we wanted to see it the +way chapters 07 and 10 taught us to: feed frames straight into the real Olympus decoder, hosted in +the debugger we built from its own DLLs, and read the PCM it produces. We rebuilt the harness to +push demuxed frames directly at the decoder's input pin. + +It didn't yield. The decoder's input frame is the manufacturer's shape, not ours — roughly **56 +bytes where our demuxer carries 42** — and whatever we fed it came back decorrelated from every +known-good reference, capping after about **25 seconds** like a trial limit we hadn't fully lifted. +We could host the decoder; we could not feed it faithfully. So we could not certify the repack — +and, per the census, it changed nothing a human ear would catch on a real file anyway. Inaudible is +inaudible. We took the demux fix and left the repack on the shelf. + +Chapter 08 would recognise the shape of that: sometimes you get the box open, and sometimes you +just learn exactly which screw you don't have. Both are worth writing down. + +## What to take away + +- **The best fix in a chapter doesn't have to be yours.** An open lock stays open because people + keep handing each other pieces. Two chapters after we thought SP demux was finished, a + contributor we already credited handed us the piece we'd skipped. +- **Measure the traffic before you chase the tail.** Four recordings a quarter is worth fixing + *because they were broken*, not because they were common — and the census is what told us which. +- **A surgical gate beats a brave rewrite.** Divert only the files that are actually broken; prove + everything else byte-identical. A decoder in production earns changes by not changing anything it + already gets right. +- **Pick the metric that's yours.** Bit-exactness against the DLL is the reference implementation's + goal. Ours is a complete, intelligible dictation. Knowing the difference is what let us take one + half of a pull request and confidently decline the other. diff --git a/docs/README.md b/docs/README.md index 19ebbf2..0ae041a 100644 --- a/docs/README.md +++ b/docs/README.md @@ -19,14 +19,19 @@ New here? Start with the [README](../README.md) for the 30-second demo and the i | 08 | [The decoder black hole](08-the-decoder-black-hole.md) | The terse engineer's handoff for the "last bug." ⚠️ **Superseded by 10 — kept as honest record.** | | 09 | [The re-sync excitation anomaly](09-the-resync-excitation-anomaly.md) | The research-paper write-up of that same "last bug": analysis-by-synthesis, nine falsified hypotheses, a hidden state machine. ⚠️ **Its central claim is wrong — read 10.** | | 10 | [The reckoning: the bug that wasn't](10-the-reckoning-the-bug-that-wasnt.md) | The twist. We built an instrumentable oracle from the vendor's DLLs, watched a reference lie to us, and settled it by **listening**. There was no bug. **The most honest chapter here.** 👂 | +| 11 | [The Grundig/Philips variant](11-the-grundig-philips-variant.md) | A real dictation whose header wasn't at `0x600` — a Grundig/Philips container the chain refused. The header-size fix that opened a whole second family of devices. 📼 | +| 12 | [Cracking the Grundig SP codec](12-cracking-the-grundig-sp-codec.md) | A Grundig Digta `.dss` that decoded to pure noise — in our pipeline, in FFmpeg, and in NCH Switch. The one nobody had decoded, cracked from the DLL. 🔊 | +| 13 | [The SP re-sync block](13-the-sp-resync-block.md) | The chapter-07 re-sync rule, never ported to the SP demuxer, comes due. Same coin: re-host the parser, read the law off the silicon, port it. 🔁 | +| 14 | [The compact block](14-the-compact-block-pause.md) | A third way DSS hides a pause — and the only one we didn't find ourselves. It arrived as a pull request. We measured, gated, shipped, and declined the half we couldn't verify. 🤝 | ## How to read it - **The integration story (00 → 05):** if you want to take an open codec into production, this is the recipe — pain, RE, integration, validation, the WASM→native call, and the lessons. - **The detective trilogy-plus (06 → 10):** if you reverse-engineer for a living, start here. Two real bugs hunted to ground (06, 07), then a rigorous investigation into a third (08, 09) that **08 and 09 get confidently wrong** — and 10, the reckoning, which is the single most useful read in the repo: how careful work fools itself, and the cheapest test that breaks the spell. +- **The relay (11 → 14):** the format keeps handing over new locks — a second device family (Grundig/Philips), a codec nobody had decoded, and a pause encoding a contributor found before we did. This is what an open format looks like when strangers keep pulling the thread. A note on chapters 08 and 09: we did not delete them when their conclusion fell. They're preserved, each behind a banner, because the *method* in them is sound and the *trap* they fell into is the lesson. A reverse-engineering log is only worth something if its dead ends stay marked. --- -*Eleven chapters from "impossible for ten years" to "production in a weekend" — including the wrong turn we're proudest of having written down. 🔓* +*Fifteen chapters from "impossible for ten years" to "production in a weekend" — including the wrong turn we're proudest of having written down. 🔓*