B6 P1: apply(text, ops), the pure operation function over an entry file's text

[markramm]

Refs #730 (B6, phase P1: the operation function, pure, not wired). ADR-0042 decisions 1, 2, 3, 6 and amendment A4.

Riskiest assumption, tested first

Assumption: ruamel's composer (YAML 1.2, A4) gives spans you can splice at. Run on ruamel 0.19.1 with a CRLF, anchor, block scalar and nested-map document:

• Holds: start_mark.index/end_mark.index are character offsets into the composed text, CRLF included. 'it''s the #1 hit' # c spans exactly the quoted scalar, and the comment sits outside it. flag: no composes as tag:yaml.org,2002:str (1.2), and d: 2026-01-15 as a timestamp.
• Breaks 1, aliases: an alias (n: *A) composes to the same node object as its anchor, so n's value span is m's bytes. A splice on n edits m. The anchor refusal therefore checks the touched key's whole subtree for any node with .anchor set. An alias shares that node, so the check catches both the anchor and its uses.
• Breaks 2, collection end marks: a block sequence's or block mapping's end_mark takes in the comment line that follows it (tags: [- x, - y] spans # comment above c\n). An unset bounded by end_mark would delete the comment above the next key. So a node's own end is computed recursively: the last child's end. For a scalar it is end_mark.
• Breaks 3, block scalar end marks: a block scalar's end_mark takes in the blank lines that follow it. This is spike 1's "unset of a folded block scalar ate the next key". It is trimmed back to the last content character. |+ keep chomping is a named case.
• Duplicate keys compose without error. They are found by walking the node tree, not by catching ruamel's load error.

So the composer works, but end_mark must not be read as "where this key's bytes end". That rule goes in a comment at the helper.

Plan

Goal, as properties. apply(text, ops, span) -> (new_text, Report) is pure:

1. Every top-level key the operations do not name keeps byte-identical own lines. The reader parses the result as exactly the operations applied to the file's parsed value.
2. The post-check runs on every call, through the reader's own parse (_frontmatter_of). If it fails, the touched top-level key is widened item-wise, copying unchanged items by bytes. If the widened result fails too, the call raises OperationRefused(reason). No existing bytes go through a serializer. New values are emitted by a small emitter that writes new lines only.
3. A replaced scalar keeps its quote style. Its trailing comment keeps its bytes, and keeps its column when the new value fits; otherwise the comment keeps its original gap (the doc's own example, title: Field notes, second draft # working title, needs the gap). A new key goes last. EOL and BOM are preserved. An emitted string that YAML 1.1 would misread is quoted, using the existing _is_yaml11_ambiguous in pyrite/utils/yaml.py (one rule, reused). Untouched yes/no/null are left alone.
4. Refused with a reason: a touched key whose subtree has an anchor or alias, any duplicate key, no YAML frontmatter (TOML +++, JSON {), and a flow-style root mapping.
5. A no-op returns text itself (result is text).
6. Operations: Set(path, value) (top level or nested: links[2].relation, params.deep.k), Append, Remove (first equal item), Unset, ReplaceBody, and AddSubkey(path, key, value), which inserts one line at the item's indent and refuses if the key is already there with a different value. The unset of a block scalar is a named test.
7. Equality is decision 3's: a str equals a date or timestamp in ISO form or at the same instant; int and float compare by value; a bool never equals an int; mappings compare unordered and lists in order.

Module: pyrite/storage/file_operations.py. It holds operations on an entry file's text, not on the model; "operations" alone would collide with the many service-level "operations" in the codebase.

Where the frontmatter ends: the caller passes it as a FrontmatterSpan(start, end, body) parameter. _frontmatter_of cannot give offsets; it returns parsed values and a stripped body. I call it read-only in the post-check, so the result is verified through the same split that reads use. The tests' span helper is the one splitter-shaped piece of code; the splitter theme repoints both.

Callers found by grep: none. Nothing is wired (P3). Footprint: 1 new module, 1 new test module and hand-written fixtures under tests/fixtures/file_operations/. No existing file is touched.

Tests:

• A fixture table of 17 hand-written shapes, crossed with the operations, under one generic oracle: untouched keys are byte-identical, the reader's parse equals the expected value, key order holds, and body and BOM are unchanged.
• Exact expected-text tests for quote style, comment column, CRLF, BOM, append style, the block-scalar unset and the empty-list edge.
• Negative tests: anchor, alias, duplicate keys, TOML, JSON, a flow root.
• A post-check guard test that injects a corrupt narrow edit and expects a widen or a refusal, never the corrupt text.
• Environment and concurrency cases do not apply to a pure function on a string (P2 has the lock).

Out of scope: wiring (P3), locks (P2), every file #708 owns, and the frontmatter splitter.

Open questions (none block this):

• The comment-column rule above. I chose "column kept when the value fits, gap kept otherwise", because the doc's example keeps the gap.
• ReplaceBody keeps the file's leading blank lines and trailing whitespace, and treats a body equal after .strip() (the reader's form) as unchanged.
• An unset leaves comment lines above the key in place ("the key's own lines").

Evidence (pushed 6717593b)

• tests/test_file_operations.py: 142 tests pass. scripts/test-affected --run at -n 4: 304 passed (14 files).
• A wider sweep, run outside the suite: 2,602 operations, each shape × {LF→CRLF, +BOM} × every key × 20 values, plus the list, nested and sub-key operations, all through the test's check_contract. 0 failures.
• ruff check pyrite/ tests/ and ruff format --check pyrite/ tests/ are clean. ruff check . still fails on dev's deploy/*/create-user.py and scripts/*appointee*.py; this branch touches neither.
• verify-red: 0 red · 142 import-only · 0 unexpected pass · 0 n/a. Why every red is import-only: the module is new and pure. No code exists without it whose behaviour a test could hit, so every test fails at import. The behavioural evidence is the mutation table below, plus test_the_oracle_catches_the_round_trip, which shows the oracle fails today's ruamel round trip (candidate A) on the hugo, yaml11-words and odd-spacing shapes.

Mutation checks (each guard removed alone, then the file restored):

Guard removed	Tests that fail
post-check (_verify returns None)	test_a_narrow_edit_that_fails_the_post_check_is_widened_not_returned, test_a_narrow_edit_that_breaks_the_value_is_widened
quote preservation (style ignored on emit)	test_a_double_quoted_scalar_stays_double_quoted, test_a_shorter_value_keeps_the_comment_column, test_a_flow_list_append_copies_its_neighbours_quoting, test_a_date_string_set_keeps_the_date_plain
anchor refusal (_refuse_anchors no-op)	test_operation_contract[anchor-alias-43], [anchor-alias-44], test_the_anchor_refusal_names_the_reason (the other two anchor cases are then refused by the post-check)
YAML 1.1 quoting (_is_yaml11_ambiguous skipped)	test_a_string_yaml11_would_misread_is_quoted_when_emitted
block-scalar end trimming	test_operation_contract[block-scalars-32], [-33], test_unset_of_a_folded_block_scalar_keeps_the_next_key, test_set_of_a_literal_block_scalar_keeps_literal_style
duplicate-key check	test_the_duplicate_key_refusal_names_the_key (added after the first run showed the loader's own error masked it)
comment column (gap only)	test_a_shorter_value_keeps_the_comment_column
widen copies unchanged items	test_a_narrow_edit_that_breaks_the_value_is_widened (asserts the hand-written item's bytes)

Covers (property 6, "the operations"; property 7, "every shape"):

Instance	Status
17 shapes of #730 section 1	guarded: test_every_shape_is_in_the_table asserts all 17 are in the table
set (top level)	guarded: table rows, test_doc_example_one_field_edit_keeps_the_comment
set a nested key	guarded: links[0].relation, params.social.mastodon, metadata.source_id (missing parents created)
append	guarded: block, flow, empty flow, mixed indents, new key, test_doc_example_append_leaves_existing_items_alone
remove an item	guarded: flow, block, the last item (test_removing_the_last_item_of_a_block_list_leaves_an_empty_list), absent (unchanged)
unset	guarded: top level, test_unset_leaves_the_comment_above_the_key, test_unset_of_a_list_does_not_take_the_comment_after_it; block scalar: test_unset_of_a_folded_block_scalar_keeps_the_next_key, ..._literal_...
replace body	guarded: leading blank line, no final newline, CRLF, BOM, the reader's form = unchanged
add a sub-key to a mapping item	guarded: test_add_subkey_inserts_one_line_at_the_items_indent, test_add_subkey_refuses_an_existing_key_with_another_value
set of a whole list or mapping	guarded: reduced to item edits by an LCS under decision 3 (test_a_set_of_a_list_is_reduced_to_item_edits)
callers	none, by design: the module is not wired (P3)

Where the frontmatter ends: the span is a parameter (FrontmatterSpan). _frontmatter_of is called read-only in _verify. The one-frontmatter-splitter theme should repoint _verify, and the test helper span_of in tests/test_file_operations.py.

🤖 Generated with Claude Code

---

Worker report (pushed 6717593)

• Evidence:
  • 142 tests pass; test-affected passes (304); CI on the pushed SHA is green.
  • A sweep outside the suite ran 2,602 operations (every shape, plus CRLF and BOM variants) with 0 failures.
  • verify-red: 0 red · 142 import-only, because the module is new. Instead there is a guard-deletion table (8 guards, each with the tests that fail) and test_the_oracle_catches_the_round_trip.
• Learned:
  • ruamel's end_mark is not where a node's bytes end: block collections run past the comment that follows, block scalars run past blank lines, and an alias shares its anchor's node.
  • ruamel 0.19.1's TimeStamp drops tzinfo under deepcopy.
  • ruamel already refuses duplicate keys.
  • On the comment column, the brief and the doc disagree. It is kept when the new value fits, and the gap is kept when it doesn't.
• Captured in: the module docstring, the _end and _scalar_edit docstrings, and named tests.
• Splitter: the frontmatter span is a parameter (FrontmatterSpan). _verify and the test helper span_of call _frontmatter_of, so the splitter theme should repoint them.
• Unsure:
  • _widen_edits is reached only by injected-fault tests.
  • |+ scalars and scalars with a leading space fall back to literal or double-quoted style.
  • The post-check does not compare a comment line between two untouched keys.
• Left: nothing in P1.

---

Fix round 1 (pushed 8fb5eea): apply() is a value-level patch

• Property: the bytes outside the named values' spans are identical. Both the module's post-check and an independent test oracle (tests/file_operations_oracle.py) compare those bytes directly. The copied _key_blocks oracle is deleted.
• Evidence:
  • CI is green.
  • 191 module tests and 353 test-affected tests pass.
  • 33 new tests fail on round 0's module, which counts as a behavioural red.
  • A new comment-dense fixture (COMMENTED) turned the old module red at once.
  • A fuzz of 1,500 malformed calls raised only OperationRefusedError.
  • Batching cut 500 sets from 30 s to 0.4 s, and a 2,000-item list from 3.2 s to 0.1 s.
• Fixed:
  • comments lost on key lines, dash-line pairs and block-scalar headers;
  • true/false/null keys, under one key-lookup rule;
  • the widen crash;
  • untyped errors;
  • |+/|2-;
  • U+2028;
  • 1e20;
  • a..b.
• Byte-ownership table and guard table: comment 5971479989.
• Unsure:
  • The oracle shares ruamel's start marks.
  • Comments between items may drop when a whole list is set (a design call).
  • A refusal limit for a literal block scalar.
• Left: repoint to split_frontmatter (refactor: one frontmatter splitter, every reader uses it #744).

[markramm]

Fix round 1, pushed at 8fb5eeac

The property. apply() is a value-level patch: every byte outside the spans of the values the operations name is identical. The module docstring states who owns which bytes. The check is independent of the edit code, and the test oracle is independent of both:

• edit: _end recurses into a value's last child;
• post-check: _named_spans walks a node's end_mark back over blank and comment-only lines;
• test oracle: tests/file_operations_oracle.py scans the lines by indentation and compares bytes with a regex fullmatch.

The old oracle _key_blocks is deleted (Andon #745).

Bytes a frontmatter holds, and their owners

Bytes	Owner	Status
key text, : and the key line's indentation	the pair (an unset)	guarded: the sweep, test_unset_finds_a_key_by_what_it_reads_as
separator after : and an inline scalar, plus the spaces after it	the value	guarded: the sweep, test_doc_example_one_field_edit_keeps_the_comment
comment after an inline value	the pair: a set keeps it, an unset removes it	guarded: test_a_shorter_value_keeps_the_comment_column, sweep (COMMENTED)
comment on a key line before a block value	the pair	fixed: test_a_block_value_set_inline_keeps_the_key_line_comment (×3), test_a_scalar_set_to_a_block_keeps_its_comment_on_the_key_line
block scalar indicators (|2-, |+)	the value	fixed: test_an_indentation_indicator_is_kept, test_keep_chomping_is_kept
comment on a block scalar's header line	the pair	fixed: test_a_block_scalar_header_comment_stays
block scalar content lines; a |+ scalar's trailing blank lines	the value	fixed: test_an_append_after_a_keep_scalar_keeps_its_blank_lines; trailing spaces on the last line are content: test_top_level_scalars_the_cold_read_crashed_on[block-scalar-trailing-spaces]
flow collection text	the value	guarded: the sweep, test_a_flow_list_append_copies_its_neighbours_quoting
block collection lines, end of key line to last content line	the value	guarded: the sweep
comment between items, inside a named list	the list (a set of the list may drop it); outside an item's unset	guarded: the sweep (COMMENTED: # between)
comment between a dash-line pair and its sibling	outside the pair's unset span	fixed: test_unset_of_a_dash_line_pair_keeps_the_comment_below_it
comment inside a nested map	the map (when named), else outside	guarded: the sweep (COMMENTED: # middle, # deep)
comment or blank lines between top-level keys	outside every span	fixed in the check: test_a_narrow_edit_touching_bytes_outside_its_span_is_never_returned (delete, rewrite, blank line), test_unset_of_the_first_key_keeps_the_comment_above_the_next
header lines before the first key; lines after the last key	outside (a new key goes after them)	guarded: the sweep (# head, # tail), test_unset_leaves_the_comment_above_the_key
delimiters, BOM	outside	guarded: check_contract, test_the_bom_survives_every_operation
body	ReplaceBody only	guarded: check_contract, test_replace_body_*

Every p6 "COMMENTS LOST" line is on a line the named value owns. Examples: - a # first under Set("tags", ["b"]) is the removed item's own line; # between under Set("tags", []) sits between items inside the set list; id: x # the id under Unset("id") is the pair's own line.

Evidence

• Red on round 0's module: 33 of the new tests fail (comments, keys, fuzz, limits, budget), and the sweep goes red once it has the comment-dense file.
• tests/test_file_operations.py: 191 passed at -n 4. scripts/test-affected --run (-n 4): 353 passed.
• The scratch sweep: 2,602 cases through the new oracle, 0 failures. The in-suite sweep covers 18 files (the 17 shapes plus COMMENTED) × {LF, CRLF, BOM}, about 4 s per variant.
• Mutations, each guard removed alone:

Guard removed	Test that fails
bytes-outside check in _try	test_a_narrow_edit_touching_bytes_outside_its_span_is_never_returned[...] (the final check then refuses)
_named_spans allows everything	the same test
final check	test_the_final_check_catches_what_the_per_operation_check_missed
key lookup rule	test_unset_finds_a_key_by_what_it_reads_as[true]
dash-line pair keeps the comment below it	test_unset_of_a_dash_line_pair_keeps_the_comment_below_it
block-to-inline keeps the key-line comment	test_a_block_value_set_inline_keeps_the_key_line_comment[...]
a |+ scalar owns its blank lines	test_keep_chomping_is_kept
anchors found before construction	test_operation_contract[anchor-alias-43]
duplicate keys compared by loaded value	test_a_segment_naming_two_keys_is_refused
- - x refusal	test_limit_a_list_written_directly_in_a_list_item_is_refused
float gets a .	test_a_large_float_reads_as_a_float_under_yaml_11_too
bare emit refuses U+2028	test_a_line_separator_is_escaped_not_written_bare
an empty path segment is refused	test_a_path_that_cannot_be_read_is_refused[a..b]
batching	test_budget_many_operations_and_a_long_list (ceiling 5 s; measured 0.4 s, about 9 s unbatched)

• Performance: 500 sets on 500 keys went from 30 s to 0.4 s, and a 2,000-item list set from 3.2 s to 0.1 s.

Learned

• ruamel's construct_document on composed nodes mutates a merge: it removes the << pair from node.value. Anchors are therefore found before construction.
• true: and 1: are duplicate keys to the loader, because True == 1 in Python.

Left

• Splitter: the frontmatter span is still a parameter. Repoint to split_frontmatter (refactor: one frontmatter splitter, every reader uses it #744): FrontmatterSpan(start=yaml_start, end=close_start, body=close_end), the _frontmatter_of call in _final_check, and the test helper span_of.
• FEEDBACK: the hallway entry moved to feedback/2026-10-03-b6-p1-operations.md (process: retro 2026-10-03 decisions (feedback/ per entry; A1 shared only when configured; quality theme) #742).

[markramm]

Fix round 2, pushed at de2672ed

The property. An edit stays narrow wherever a narrow edit can express it. Kept values keep their bytes, including comments between unchanged pairs and items (decision 1). A widen copies unchanged items' bytes and the lines between them, in the file's order (decision 2). Every failure is OperationRefusedError.

Fixed:

• B1: _edit_map now finds new keys by value under the key rule (_exact(kept, k)), not by id().
• B2: the widen rebuilds in the file's order (_widen_map, _widen_seq), copying kept pairs and items with the lines between them. It is checked against the whole top-level value it rebuilds.
• B3: a complex key anywhere is refused with a reason.
• L4: tagged values compare equal after a reparse (TaggedScalar, sets, bytes).
• L1: each operation is checked against the text it applied to, so same-key sequences work.
• L2: _unshare breaks sharing in the touched value.
• L5: a key starting with --- or ... is quoted.

Stated and pinned: L3 (a batch indents new lines by the conventions before it), L6 (- # comment), and comment-looking lines inside literal and multi-line quoted scalars.

Evidence:

• Behavioural reds: 21 of the new tests fail on round 1's module.
• Suite: tests/test_file_operations.py 220 passed at -n 4; scripts/test-affected --run 640 passed.
• Probes on the new module:
  • p11: 3,765 ok.
  • p18: 6,000 calls, 0 crashes (it was 23).
  • p19: 2,986 same; the 14 differences are all L3.
  • p10: the one flag is "unset tags[0] twice". By the span rule, the second unset removes the only remaining item, so it is a set of the list to [] and owns the list's span. The probe's span was narrower.
• Narrow: every sweep edit asserts widened == (), across 18 files × LF/CRLF/BOM. The narrow-supported edits are the sweep's whole operation list: set (scalar, list, map, type change), unset (key, item, nested key), append, remove, add-sub-key, and nested set through missing parents.

Mutations, each guard removed alone:

Guard removed	Test that fails
keys compared by value (B1)	test_a_map_set_with_long_keys_stays_narrow, test_an_item_map_set_keeps_its_comments
widen keeps lines between pairs	test_the_widen_of_a_map_keeps_the_files_order_and_its_comments
widen keeps lines between items	test_the_widen_of_a_list_keeps_the_comments_between_kept_items
widen owns the top-level value	test_the_widen_of_a_nested_operation_rebuilds_its_top_level_value
complex key refused	test_a_complex_key_anywhere_is_refused_with_a_reason
TaggedScalar in _plain	test_a_tagged_value_elsewhere_does_not_block_an_edit[!custom 1]
values_equal fallback	the same test, [!!set] and [!!binary]
_unshare	test_a_shared_value_in_one_call_does_not_alias_in_the_model
marker key quoted	test_a_new_key_spelled_like_a_document_marker_is_quoted
final check by the reader	test_the_final_check_catches_what_the_per_operation_check_missed

Removed: the final check's byte replay. It repeated _try's check exactly, so deleting it failed no test.

Left:

• Repoint to split_frontmatter (refactor: one frontmatter splitter, every reader uses it #744): the span parameter, _final_check's _frontmatter_of, and the test helper span_of.

[markramm]

Correction to the round-2 comment: tests/test_file_operations.py is 219 passed at -n 4, not 220.