Skip to content

add first-class yaml support to the standard library - #628

Merged
kacy merged 5 commits into
mainfrom
yaml-support
Jul 31, 2026
Merged

add first-class yaml support to the standard library#628
kacy merged 5 commits into
mainfrom
yaml-support

Conversation

@kacy

@kacy kacy commented Jul 31, 2026

Copy link
Copy Markdown
Owner

summary

adds std.yaml, a yaml parser for the subset configuration files actually
use, built to the same shape as std.json and std.toml: a threadlocal node
pool addressed by opaque Int handles, type_of for the kind, the toml
accessor names, open_scope/close_scope with the pool_mark_*/pool_restore
pair underneath, and the full generic decode[T] family.

what parses: block mappings and sequences nested in each other, the compact
- name: a form, sequences written at their key's own indentation, flow
collections, plain scalars resolved against the yaml 1.2 core schema, single-
and double-quoted scalars with escapes including \uXXXX, comments, literal
and folded block scalars with strip and keep chomping, and multiple documents
separated by --- (parse returns the first, parse_all returns them all).

what does not, each refused with a message that names it rather than being
quietly mis-parsed: anchors and aliases, merge keys, tags, complex keys,
directives, multi-line plain scalars, explicit block-scalar indentation
indicators, flow collections spanning lines, and content on a --- line.
anchors are the interesting exclusion. alias expansion is the billion-laughs
amplification vector, and a depth cap does not help because the blowup is in
width, so doing them safely needs an expansion budget, which is a different
design. tabs in indentation are refused too; that one is yaml's own rule.

six limits bound everything the input controls: 4 MiB per document (the same
ceiling std.net.grpc puts on a message), 64 levels of nesting, 262144 nodes,
65536 entries per collection, 1 MiB per scalar and 4096 bytes per key. they are
module constants, not a table built on first use, because a lazily initialized
table is a race under the green runtime.

typed decoding needed a small compiler change. std.toml and std.yaml parse
into the same thing, a node pool reached through require_string,
require_int, require_bool, has, require_table and three pool marks, so
the emitter already knew how to lower a decode against that shape. it just
called the shape "toml". this renames the family to handle-pool decode, makes
the last three hardcoded std_toml_ names follow the call's receiver like
every other name in that path already does, and adds std.yaml to the set.
toml's private toml_require_table becomes a public require_table so both
modules answer to one member name. the bootstrap seed is regenerated.

what was tested

  • make test: all fourteen steps, run in slices because the whole target
    outruns a single command window. 100 examples, 297 regression cases native
    and again through the self-hosted compiler, invalid parse and checker cases,
    cli regressions, the ir contract checks, 15 parity examples, docsite and
    sitegen goldens.
  • verify-green-corpus and verify-osthread-corpus: 297 cases each, green at
    default and at one worker, and again with PITH_GREEN=0. the 100 examples
    also run clean under PITH_GREEN=0.
  • make bootstrap-ir-checks-only and make run-examples-self-only, since the
    emitter changed: ir invariants, the ir fixed point on all ten sources, and
    the self-hosted example corpus.
  • make test-std-self-only: 76 std modules, including the five colocated tests
    in std/yaml.pith.
  • make leak-check: the new leak_yaml_parse case grew 48kb across 200k and
    800k rounds, against a 2048kb limit.
  • make memcheck: test_yaml_structure, test_yaml_malformed and
    test_yaml_derived_decode join the valgrind set and are clean.

the malformed suite is 26 cases and each one asserts both that the parse failed
and that the message says why: unterminated quotes in a value, in a key and in
a single-quoted scalar; a bad escape and a truncated \u; tab indentation;
three kinds of indentation error; truncated flow sequences and mappings; a flow
entry with no value; trailing junk after a flow collection; each of the eight
refused features; a 400-level indentation bomb and a 512-bracket flow bomb,
both stopping at the depth cap; a 16 KiB key; and a 5 MiB document. a legal
40-level document still parses, and the parser works normally afterwards.

notes

two pre-existing leaks turned up while building the leak case. neither is
introduced here and neither is fixed here:

  • get_string on a handle-pool module leaks a string per call, roughly 31
    bytes a round. std.toml measures identically, so it is not yaml-specific.
  • decode_text[T] leaks roughly 280 bytes a round, again identically for
    std.toml. it looks like the known result-local leak: the lowered body
    stores a parse_required result in a temp and reads .ok off it.

the leak case therefore exercises the scoped parse paths, which are flat.

one thing worth knowing for anyone writing a std module: assigning a fresh list
literal to a threadlocal mut xs: List[T] leaks the old list, about 800 bytes
a round in this parser's case. clearing in place avoids it, which is what
scan_lines does now.

kacy added 5 commits July 31, 2026 03:28
std.yaml parses the part of yaml 1.2 that configuration files actually
use into the same handle-and-node-pool shape std.json and std.toml
already have: block mappings and sequences, flow collections, plain,
single- and double-quoted scalars, literal and folded block scalars with
chomping, comments and multiple documents.

everything outside that subset is refused with a message rather than
guessed at, and every input-driven quantity is bounded. anchors and
aliases are the notable exclusion: alias expansion is the billion-laughs
amplification vector, and a config parser that expands them hands an
attacker a denial of service.
covers scalar typing and quoting, nested block structures, flow
collections, block scalars and chomping, multiple documents, node-pool
scopes, and a parity case that reads the same json document through
std.json and std.yaml.

the malformed suite is the important half: unterminated quotes, tab
indentation, truncated flow collections, nesting bombs written both as
indentation and as brackets, an enormous key and an oversized document
each have to come back as a clean error.
std.toml and std.yaml parse into the same thing: a node pool addressed
by Int handles, reached through require_string, require_int,
require_bool, has, require_table and three pool marks. the compiler
already knew how to lower a decode against that shape; it just called
the shape 'toml'.

this renames the family to handle-pool decode, makes the last three
hardcoded std_toml_ names follow the call's receiver instead, and adds
std.yaml to the set. toml's private toml_require_table becomes a public
require_table so both modules answer to one member name.

decode, decode_text, decode_path, decode_text_path, decode_file and
decode_file_path now all work against yaml, with optional fields,
defaults and nested structs behaving as they do for toml.
a scoped parse loop and a scoped multi-document parse loop both have to
stay flat across round counts, and three of the yaml cases join the
valgrind set because a parser allocates heavily enough that an ownership
mistake shows up there first.

the line table is now cleared in place instead of being rebuilt: handing
the allocator three fresh lists per parse showed up as roughly 800 bytes
a round in the growth gate.
docs/yaml.md writes out what parses, what is refused and why, the six
input-driven limits, and the node-pool scope rules. examples/yaml_ops
runs the same ground as examples/toml_ops so the two read side by side,
and joins the native/self-host parity set.
@kacy
kacy merged commit 2d1f786 into main Jul 31, 2026
2 checks passed
@kacy
kacy deleted the yaml-support branch July 31, 2026 06:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant