Load a file written against an earlier format version

changelog/43.added.md

@@ -0,0 +1 @@
+A file written against an earlier format version now loads, in both text formats. A release that changes a syntax registers one migration under its own version with `qp.register_migration`, and reading applies every migration between the version in the file's header and the running one, oldest first, to the lines in memory — the file on disk is never rewritten, and writing it back out writes today's version. A release that breaks nothing registers nothing, so the chain carries an entry per breaking change rather than per release, and an older file with nothing behind it loads as it is. A migration rewrites lines one for one and may not add or drop any, which is what keeps a `ParseError`'s line number and every `source_map` entry naming a line of the file its author opened; one that breaks that count raises `ValueError` and names itself. `.qp` and `.wfl` share the version scale, since each header carries the library version cut to `major.minor`, but not their rewrites: `file_format="qp"` or `"wfl"` picks the table, and a change to vocabulary the two files share is one rewrite registered twice. `qp.serialization.known_migrations` lists what is registered for a format. A vendor extension gets the same mechanism for the wire form of its own operations through `qp.register_vendor_migration`, keyed to the extension's version: the `require <vendor> <major.minor>` line says which release wrote the body, and the extension's rewrites carry it up to the installed one, so renaming an operation no longer orphans the files a user already has.

changelog/43.changed.md

@@ -0,0 +1,3 @@
+Both readers now take a header version to be exactly `major.minor`, and refuse any version above the running one. A file from a later release used to load when its major matched, on the grounds that a minor only ever adds; it is refused now, because a reader has no way to know what a later release changed and no migration runs backwards. A header carrying a patch (`#!QProgram 0.2.3`) or a bare major (`#!QProgram 0`) is refused as well: a patch release changes code and never the format, so a file has no patch to declare. The `.qp` grammar's header terminal already said as much, and the hand-written parser now agrees with it.
+
+A `require <vendor>` line is read by that same rule against the extension it names, which changes two things and merges the two failures into one message. A line carrying a patch (`require myvendor 0.1.9`) is refused rather than rounded down, and a line naming an earlier major is migrated rather than refused, so `major versions must match` and `minor version too old` are replaced by `file requires myvendor 0.7, newer than the installed myvendor 0.1.0 — install myvendor 0.7 or newer`. What an extension registers with `register_vendor_version` is unaffected: that is a package version, and its patch component is still read and ignored.

docs/developer/architecture.md

@@ -26,7 +26,7 @@ qprogram/
 │   ├── conftest.py                 # shared schema / program / waveform fixtures
 │   └── _dummy_vendor.py            # a complete in-tree vendor extension, used as a fixture
 └── src/qprogram/
-    ├── __init__.py                 # the public surface: 105 names, parser entry points lazy
+    ├── __init__.py                 # the public surface: 107 names, parser entry points lazy
     ├── py.typed                    # PEP 561 marker: the package ships its own annotations
     ├── qprogram.py                 # QProgram builder, control-flow contexts, vendor registry
     ├── buses.py                    # BusSchema, BusRef, BusNaming, typed presets
@@ -61,6 +61,7 @@ qprogram/
         ├── parser.py               # loads / load
         ├── registry.py             # registry-driven dispatch
         ├── _specs.py               # per-op serialize/parse callbacks
+        ├── migrations.py           # rewrites that load a file from an older version, core or vendor
         └── _format.py              # the .qp format version constant
 ```
 

docs/developer/contributing.md

@@ -191,8 +191,8 @@ Use this when you are not sure which file to touch.
 | New sweep source | `src/qprogram/sweeps/builtin.py` or `combinators.py`, exported from `sweeps/__init__.py`, and added to the `register_sweep_source` loop in `src/qprogram/serialization/_specs.py`. That call also registers the class's `TOKEN` with the capability registry. |
 | Parser change | `src/qprogram/serialization/parser.py`. |
 | Writer change | `src/qprogram/serialization/writer.py`. |
-| Grammar change | `src/qprogram/grammar/qp.lark`, kept in step with the parser by `tests/test_grammar.py`. |
-| New vendor operation | The vendor's own package. See [Building a vendor extension](vendor-extensions.md). |
+| Grammar change | `src/qprogram/grammar/qp.lark`, kept in step with the parser by `tests/test_grammar.py`. A change that stops an existing file from parsing also needs a [migration](serialization-internals.md#adding-a-migration) in `src/qprogram/serialization/migrations.py`, registered under the version that ships it, so files written before it keep loading. |
+| New vendor operation | The vendor's own package. See [Building a vendor extension](vendor-extensions.md). A change to an existing operation's wire form also needs a [`register_vendor_migration`](vendor-extensions.md#keeping-older-files-loading) in that package, registered under the version that ships it. |
 | New vendor package | A separate package depending on `qprogram`. Same guide. |
 | Docs | `docs/`, with the nav in `zensical.toml`. |
 | Changelog entry | One fragment in `changelog/`. Never edit `CHANGELOG.md` by hand. |

docs/developer/serialization-internals.md

@@ -29,10 +29,159 @@ metadata falls back to `"0.0"`. The `.wfl` format's
 `WAVEFORM_LIBRARY_FORMAT_VERSION` is derived the same way, which is why the two
 headers carry the same number.
 
-The version is emitted in the `#!QProgram` header and checked on load. Only the
-major component is binding: a file whose major differs is rejected with
-`Unsupported format version`, and any minor within the same major loads, so a
-`0.4` file opens under a `0.2` runtime.
+The version is emitted in the `#!QProgram` header and checked on load. A file
+from a later release is rejected with `Unsupported format version`, whichever
+component moved; an older file is migrated up to this version. The header
+carries `major.minor` exactly, since a patch release cannot change the format,
+and a `require <vendor>` line is read the same way against its extension's
+version.
+
+## Migrations
+
+`src/qprogram/serialization/migrations.py` holds the rewrites that let today's
+parser read yesterday's syntax. Each one is registered under the version that
+broke something:
+
+```python
+_SWEEP_KEYWORD = re.compile(r"(?<=^  )sweep\b")
+
+
+@register_migration("0.3")
+def _sweep_became_for(lines: list[str]) -> list[str]:
+    return [_SWEEP_KEYWORD.sub("for", line) for line in lines]
+```
+
+`_parse_header` reads the version off the header and, when the file is older
+than the running one, replaces the parser's lines with the result of running
+every migration in `(file version, running version]`, oldest first. Nothing
+else in the parser knows a migration happened, and the file on disk is never
+touched: the rewrite lives as long as the parse. `WaveformLibrary.loads` does
+the same through `_migrated` in `src/qprogram/waveform_library.py`, against the
+`"wfl"` table.
+
+Three rules make that safe to rely on:
+
+- **One migration per breaking change, not per release.** A release that leaves
+  the syntax alone registers nothing, and a file two releases behind collects
+  every step in between. This is why the registry is a sorted list of steps
+  rather than a chain of parent revisions: there is no node to write for a
+  quiet release.
+- **Lines in, as many lines out.** A migration may rewrite a line, and may look
+  at its neighbours, but may not add or drop one. That is what keeps a
+  `ParseError`'s line number and every `source_map` entry naming a line of the
+  file its author opened. `migrate_lines` checks the count and raises
+  `ValueError` naming the migration that broke it. A change that genuinely
+  needs to restructure lines is the point at which this mechanism grows a line
+  map; until then the invariant is worth more than the flexibility.
+- **The header is not a migration's business.** The rewrite is handed every line
+  including the header, so the indices line up with the file, but the reader has
+  already read the version off it and moves past it.
+- **One table per format, one version scale for both.** `FORMAT_VERSION` and
+  `WAVEFORM_LIBRARY_FORMAT_VERSION` are the same library version cut the same
+  way, so `_RUNNING_VERSION` bounds both chains and a release's breaking change
+  carries the same number in either file. The rewrites stay apart, since
+  `"pi" = Square(...)` in a library and `play "drive" Square(...)` in a program
+  are not the same text; a change to the vocabulary they do share is one
+  function registered under both formats, which is what the stacked decorator
+  in `register_migration`'s docstring shows.
+
+A migration registered under a version that has not shipped yet is skipped,
+since the runner only applies steps up to the running version. That makes it
+safe to write the migration in the same commit as the change that needs it,
+before the release is cut.
+
+### Adding a migration
+
+Say 0.3 renames the `Rectangular` waveform to `Square`. Waveform constructors
+are keyed by class name in the registry, so one rename changes how a pulse is
+spelled in a program body and in a library entry at once, and both of these are
+files somebody already has on disk:
+
+<!-- check: skip -->
+```
+#!QProgram 0.2
+
+body:
+  play "drive_q0" Rectangular(amplitude=0.5, duration=200)
+```
+
+```
+#!WaveformLibrary 0.2
+"pi_pulse" q[0].drive = Rectangular(amplitude=0.5, duration=200)
+```
+
+The rewrite goes at the bottom of
+`src/qprogram/serialization/migrations.py`, below the runners, so that
+registration happens on the import both readers already perform. The module
+imports no `re` today, so the first migration brings it:
+
+```python
+# src/qprogram/serialization/migrations.py
+_RECTANGULAR = re.compile(r"\bRectangular\(")
+
+
+@register_migration("0.3")
+@register_migration("0.3", file_format="wfl")
+def _rectangular_became_square(lines: list[str]) -> list[str]:
+    """Rewrite the constructor 0.3 renamed, in a program body and in a library entry alike."""
+    return [_RECTANGULAR.sub("Square(", line) for line in lines]
+```
+
+Two decisions are worth spelling out. The decorator is stacked because a
+waveform constructor is vocabulary the two formats share; a change to the body
+grammar, a block header or an operation keyword, is `"qp"` alone, and the
+`"alias" coord = waveform` entry line is `"wfl"` alone. The pattern is anchored
+because a migration is text in and text out with no parse in between:
+`\bRectangular\(` matches the constructor call and nothing else on the line,
+leaving a waveform aliased `"Rectangular"` and a program label reading
+`Rectangular pulse calibration` alone, where a pattern matching the bare word
+would rewrite both.
+
+The rewrite earns two tests in two places, because `tests/test_migrations.py`
+empties both tables around every test so that its fixtures can register
+throwaway rewrites without the shipped chain running underneath them. That puts
+the registered chain out of reach there, so what that file tests is the function
+itself, called on the lines it would be handed:
+
+```python
+# tests/test_migrations.py
+def test_rectangular_became_square_leaves_an_alias_of_the_same_name_alone():
+    entry = '"Rectangular" q[0].drive = Rectangular(amplitude=0.5, duration=200)'
+    assert _rectangular_became_square([entry]) == ['"Rectangular" q[0].drive = Square(amplitude=0.5, duration=200)']
+```
+
+Whether it runs on the file that needs it is the other half, and belongs
+wherever the registered chain is left in place, such as
+`tests/test_serialization.py`:
+
+```python
+# tests/test_serialization.py
+def test_a_file_written_before_the_waveform_rename_still_loads():
+    text = '#!QProgram 0.2\n\nbody:\n  play "drive_q0" Rectangular(amplitude=0.5, duration=200)\n'
+    assert isinstance(qp.loads(text).body.elements[0].waveform, qp.waveforms.Square)
+```
+
+The version there is written out rather than taken from `tests/_header.py`,
+since the point of the test is that one particular older version still loads. It
+starts passing when 0.3 is cut: until then the file it writes carries the
+running version rather than an earlier one, so nothing runs and the assertion
+fails. That is the one claim about a migration its own release has to make true,
+which is why the unit test above it is what guards the rewrite in review.
+
+### Vendor migrations
+
+One level down, a `require <vendor>` line has the same problem: an extension
+that renames an operation orphans the files its users already have.
+`register_vendor_migration(vendor, version)` is the same mechanism against that
+extension's version. `_check_vendor_compat` runs the vendor's chain over the
+lines when the line asks for an earlier release than the one installed, which is
+why an earlier vendor major is no longer refused. The ceiling there is the
+installed extension, not the library, so `migrate_vendor_lines` takes it as an
+argument rather than reading `_RUNNING_VERSION`. `.wfl` files declare no vendor,
+having no `require` line, so vendor tables are consulted for `.qp` only. The
+same worked example from inside an extension package, registration site and test
+included, is under [keeping older files
+loading](vendor-extensions.md#keeping-older-files-loading).
 
 ## The registries