Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 18 additions & 6 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,11 +11,11 @@ Everything else exists today.
| Component | Responsibility |
| --------- | -------------- |
| `converter/cli.py` | Argument parsing, target-format selection, the interactive prompt, usage errors, exit codes |
| `converter/profiles.py` | One declarative profile per target format: the copy mask, the fallback encoder and the drop reason per stream type, the container flags, the ffmpeg muxer that writes it, the cheap and last-resort attempts the format declares as data, and whether that cheap attempt's mapping is partial by construction |
| `converter/jobs.py` | The generic conversion engine: turns a profile plus a probed stream list into an ordered ladder of attempts, and into the notes a *successful* partial cheap attempt owes. It owns the *order* of the rungs and how the selective rung is built; the profiles own what each declared rung contains |
| `converter/profiles.py` | One declarative profile per target format: the copy mask, the fallback encoder and the drop reason per stream type, the container flags, the ffmpeg muxer that writes it, the cheap and last-resort attempts the format declares as data, whether that cheap attempt's mapping is partial by construction, and — for a probe-first profile — which streams it writes as sidecar files instead (codecs, options, muxer, suffix) |
| `converter/jobs.py` | The generic conversion engine: turns a profile plus a probed stream list into an ordered ladder of attempts, and into the notes a *successful* partial cheap attempt owes. It owns the *order* of the rungs, how the selective rung is built and which streams a sidecar step writes; the profiles own what each declared rung contains |
| `converter/batch.py` | Bounded parallel execution, the per-file outcome, the progress bar, the aggregate summary and the process exit code; hands each result to a caller-supplied callback |
| `converter/report.py` | Rendering results: the human text lines and the `--json` records (JSON Lines), both over `batch.Result` / `batch.Summary` |
| `converter/paths.py` | Input discovery, output-path construction, tree mirroring, collision detection, Windows path-length diagnosis |
| `converter/report.py` | Rendering results: the human text lines and the `--json` records (JSON Lines, including each converted file's `sidecars`), both over `batch.Result` / `batch.Summary` |
| `converter/paths.py` | Input discovery, output-path construction, sidecar naming (language normalisation, the duplicate rule, the stale-partial sweep), tree mirroring, collision detection, Windows path-length diagnosis |
| `converter/ffmpegtool.py` | Locating ffmpeg and ffprobe, building argv, running without a shell, probing streams; keeps every running process in a registry it can terminate, bound to a Job Object on Windows |
| `converter/__main__.py` | The `python -m converter` entry point |

Expand Down Expand Up @@ -95,6 +95,14 @@ only on `cli`.
takes. A probe-first profile (`web`) has no cheap attempt at all and succeeds
on the selective rung instead; see the 2026-08-26 (issue #41) entries in
`docs/specs/archive/spec-profile-registry.md`'s Decision log.
A profile that routes streams to sidecars (`web`'s text subtitles) adds one
step after whichever rung wrote the output: `jobs.plan_sidecars` names the
streams from the stream list already probed, `paths.sidecar_paths` names
their files, and `batch` runs **one** further ffmpeg process with one
`.partial` output per sidecar. A failure there drops the sidecars with a
note each and never the output. Every sidecar is renamed into place
**before** the output, so an output at its final path implies its sidecar
step finished (`docs/specs/spec-subtitle-sidecars.md`).
2. **Degradation.** The attempt exits non-zero, so *now* `ffmpegtool.probe_streams`
describes the file. Each stream is first resolved to a rule — by its
disposition when it is an attached picture and the profile declares a rule for
Expand All @@ -121,7 +129,10 @@ only on `cli`.
3. **Idempotent re-run.** An output that already exists and no `--overwrite` makes
the file `skipped` without starting a process (a stale `<output>.partial` from
a killed run is removed first), so a second run over a finished
tree does no work for the files it already converted. A source whose output
tree does no work for the files it already converted. Sidecars take no part
in that decision: the output's existence alone decides it, and stale sidecar
partials beside it are swept by name grammar (`paths.stale_sidecar_partials`)
whether the file is skipped or not. A source whose output
path would be its own input path is `skipped` too — reported rather than passed
over in silence, and counted, per `docs/design/source-selection.md`.
4. **Failure.** The `.partial` file is removed — an existing output is left
Expand All @@ -142,7 +153,8 @@ only on `cli`.
thread (`KeyboardInterrupt`, or `ffmpegtool.Terminated` from the SIGTERM
handler), which calls `ffmpegtool.terminate_all`: the registry closes to new
spawns and every running process is killed and reaped. Each worker sees the
shutdown flag after its `run()` returns, deletes its own `.partial` and stops
shutdown flag after its `run()` returns, deletes its own `.partial` (and its
sidecar partials, when the sidecar step was running) and stops
without a result; the main thread waits for them, then removes any partial
still in flight. The run exits 130 or 143 without a summary. A Windows
`TerminateProcess` cannot be caught: the Job Object kills ffmpeg with the
Expand Down
22 changes: 20 additions & 2 deletions docs/design/degradation-ladder.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,8 +24,11 @@ flowchart TD
PLAN["build the selective plan<br/>(no subprocess — see stream-decision.md)"]
SEL["Attempt 2 — selective<br/>(ffmpeg)"]
FIN["Attempt 3 — the profile's last-resort attempt<br/>(ffmpeg; a profile may declare none)"]
OK["converted — the winning attempt's notes are reported"]
BAD["failed — partial output removed, ffmpeg's stderr kept per rung"]
OK["the attempt succeeded — its notes are the run's notes"]
SCQ{"did the stream plan route any stream to a sidecar?<br/>(only a probe-first profile can declare one)"}
SCS["sidecar step — one ffmpeg, one output per sidecar<br/>(each to its own .partial; a failure drops the sidecars<br/>with a note each, never the output)"]
DONE["rename into place — sidecars first, the output last;<br/>converted, the notes are reported"]
BAD["failed — partial output removed, the error kept<br/>(stderr per rung, or the rename's reason)"]

PF -->|"no"| A
PF -->|"yes — no cheap attempt; the one probe comes first"| P
Expand All @@ -47,10 +50,25 @@ flowchart TD
SEL -->|"exit != 0"| FIN
FIN -->|"exit 0"| OK
FIN -->|"exit != 0, or no last-resort attempt declared"| BAD
OK --> SCQ
SCQ -->|"no"| DONE
SCQ -->|"yes"| SCS
SCS --> DONE
DONE -->|"the output's own rename fails —<br/>renamed sidecars stay"| BAD
```

## Rules the diagram encodes

- **The sidecar step runs after the ladder, never inside it.** Whichever rung
wrote the output — the selective rung or the last resort — the streams the
plan routed to a sidecar (`stream-decision.md`'s `SIDECAR`) are written by one
further ffmpeg process with one output per sidecar, built from the stream
list the up-front probe already holds, so it spends no probe. A sidecar that
fails costs its own file and a note, never the output and never a rung:
carrying sidecars as extra outputs on a rung would let a subtitle the WebVTT
encoder rejects push the video down to a re-encode. Every sidecar is renamed
before the output, so an output at its final path implies its sidecar step
finished (`docs/specs/spec-subtitle-sidecars.md`).
- **A probe-first profile starts at `P`.** It declares no cheap attempt, because a
blind remux is precisely what would carry streams the target accepts but cannot
use — `web`'s HEVC or AC-3 in an MP4 a browser cannot play. Its one probe is the
Expand Down
16 changes: 16 additions & 0 deletions docs/design/source-selection.md
Original file line number Diff line number Diff line change
Expand Up @@ -140,6 +140,22 @@ flowchart TD
file is counted, because `0 converted, 12 skipped` is the idempotent-re-run
evidence the vision promises. A `.txt`, or a file inside the output tree, is not,
because counting everything a directory happens to hold means nothing.
- **Every path a source writes is guarded, but only the output is selected.**
A target that writes sidecars (`--to web`'s WebVTT files) has more than one
path per source, and selection cannot know them: their names follow from the
source's subtitle streams, which only a probe reveals. The guards above still
cover them, by construction of the name — a sidecar is the output's name with
its suffix replaced by `.<lang>[.<k>]<sidecar suffix>`, where `<lang>` is never
purely numeric, `<k>` always is, and neither holds a dot. Two distinct output
names in one directory therefore never share a sidecar (their components
cannot line up), a sidecar never equals an output (different suffix), and it
never equals a selected source's input path (the sidecar suffix is in no
source-suffix set, and a named file's own sidecars carry one component more
than its stem). So `COLL`, `SELF` and `HAZ` hold for sidecars without a node of
their own, and `cli.py` is unchanged. `EXISTS` is the one guard that is asked
**per sidecar, after the ladder**, not here: an existing sidecar path is left
alone without `--overwrite` and named in a note, and the output's own
existence alone decides the skip (`docs/specs/spec-subtitle-sidecars.md`).
- **Selection cannot tell whether a source can *produce* the target.** Whether a
video-only file has audio to put in a WAV is only knowable from a probe, and
selection has no probe to spend: the one the ladder may now spend on a
Expand Down
31 changes: 21 additions & 10 deletions docs/design/stream-decision.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,8 @@

> Design artifact (`docs/design.md`, `kind: concept`). The sibling of
> `degradation-ladder.md`: that file decides the order of attempts, this one
> decides what happens to a single stream inside the selective rung.
> decides what happens to a single stream inside the selective rung — including
> whether it leaves the rung for a sidecar file written after the ladder.
>
> Declared deviation from `docs/design.md`'s per-stream convention: the contract
> asks for one decision node per stream type. A profile-driven engine has no
Expand All @@ -13,11 +14,12 @@
## The decision this settles

Given one probed stream and the target profile's rule for that stream's type,
whether the stream is copied, re-encoded, or dropped — and what the resulting
note says. No node here spends a subprocess call: the whole plan is built in
Python from one stream list. `TARGET` below is the profile's display label
(`MP4`, `WAV`), `TARGET_CODEC` the human-readable name of what the fallback
encoder produces (`h264`, `aac`), `DROP_REASON` the reason the rule declares.
whether the stream is copied, re-encoded, dropped, or written as a sidecar — and
what the resulting note says. No node here spends a subprocess call: the whole
plan is built in Python from one stream list. `TARGET` below is the profile's
display label (`MP4`, `WAV`), `TARGET_CODEC` the human-readable name of what the
fallback encoder produces (`h264`, `aac`), `DROP_REASON` the reason the rule
declares.

```mermaid
flowchart TD
Expand All @@ -35,6 +37,8 @@ flowchart TD
D1["drop — note: t stream i (c) dropped: not supported by TARGET"]
D2["drop — note: t stream i (c) dropped: TARGET holds LIMIT t stream<br/>(the noun agrees in number with LIMIT)"]
D3["drop — note: t stream i (c) dropped: DROP_REASON"]
SIDE{"does the rule declare a sidecar<br/>that accepts c?"}
SIDECAR["sidecar — not mapped into the output; written to a file<br/>of its own after the ladder (degradation-ladder.md's sidecar step)<br/>note only when the sidecar format gives up styling"]

S --> PIC
PIC -->|"yes — use that rule"| ROOM
Expand All @@ -46,7 +50,9 @@ flowchart TD
MASK -->|"yes"| COPY
MASK -->|"no"| ENC
ENC -->|"yes"| REENC
ENC -->|"no"| D3
ENC -->|"no"| SIDE
SIDE -->|"yes"| SIDECAR
SIDE -->|"no"| D3
REENC --> OPT
OPT -->|"yes"| OVERRIDE
OPT -->|"no"| STAND
Expand All @@ -61,9 +67,14 @@ flowchart TD
declares no `attached_pic` rule is unaffected, since the fallback is what it
already did. The engine still counts a carried picture under `video`, because
ffmpeg numbers it as a video output stream.
- **Three outcomes, never a fourth.** A stream is accepted, re-encoded, or
dropped. Every drop edge names the reason, because a silent drop is exactly
what the vision forbids.
- **Four outcomes, never a fifth.** A stream is accepted, re-encoded, dropped,
or — only where the rule declares a sidecar (`web`'s text subtitles) — moved
into a sidecar file of its own. `SIDE` sits on `ENC`'s `no` edge, so a stream
the output itself can carry is never diverted: the sidecar is the alternative
to a drop, not to a copy (`docs/specs/spec-subtitle-sidecars.md`). A sidecar
is not a drop and earns no drop note; a sidecar format that cannot hold the
stream's styling earns a styling note instead. Every drop edge names the
reason, because a silent drop is exactly what the vision forbids.
- **A re-encoded stream may also need a source-dependent option.** `OPT` sits
after `REENC`: a rule may declare a value its fallback needs only when a
second, independent probed property — not the one `MASK` already used to
Expand Down
2 changes: 1 addition & 1 deletion docs/roadmap.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@
| 11 | json-output | [spec-json-output.md](specs/archive/spec-json-output.md) | [#11](https://github.com/bhemsen/converter/milestone/11) |
| 12 | abort-safe-writes | [spec-abort-safe-writes.md](specs/archive/spec-abort-safe-writes.md) | [#12](https://github.com/bhemsen/converter/milestone/12) |
| 13 | web-target | [spec-web-target.md](specs/archive/spec-web-target.md) | [#13](https://github.com/bhemsen/converter/milestone/13) |
| 14 | subtitle-sidecars | — | — |
| 14 | subtitle-sidecars | [spec-subtitle-sidecars.md](specs/spec-subtitle-sidecars.md) | [#14](https://github.com/bhemsen/converter/milestone/14) |

A phase gets a Spec link once `/plan` drafts it, and a Milestone link once the
spec is merged. The milestone (open/closed + issue progress) is where status
Expand Down
Loading
Loading