add first-class yaml support to the standard library - #628
Merged
Conversation
std.yaml parses the part of yaml 1.2 that configuration files actually use into the same handle-and-node-pool shape std.json and std.toml already have: block mappings and sequences, flow collections, plain, single- and double-quoted scalars, literal and folded block scalars with chomping, comments and multiple documents. everything outside that subset is refused with a message rather than guessed at, and every input-driven quantity is bounded. anchors and aliases are the notable exclusion: alias expansion is the billion-laughs amplification vector, and a config parser that expands them hands an attacker a denial of service.
covers scalar typing and quoting, nested block structures, flow collections, block scalars and chomping, multiple documents, node-pool scopes, and a parity case that reads the same json document through std.json and std.yaml. the malformed suite is the important half: unterminated quotes, tab indentation, truncated flow collections, nesting bombs written both as indentation and as brackets, an enormous key and an oversized document each have to come back as a clean error.
std.toml and std.yaml parse into the same thing: a node pool addressed by Int handles, reached through require_string, require_int, require_bool, has, require_table and three pool marks. the compiler already knew how to lower a decode against that shape; it just called the shape 'toml'. this renames the family to handle-pool decode, makes the last three hardcoded std_toml_ names follow the call's receiver instead, and adds std.yaml to the set. toml's private toml_require_table becomes a public require_table so both modules answer to one member name. decode, decode_text, decode_path, decode_text_path, decode_file and decode_file_path now all work against yaml, with optional fields, defaults and nested structs behaving as they do for toml.
a scoped parse loop and a scoped multi-document parse loop both have to stay flat across round counts, and three of the yaml cases join the valgrind set because a parser allocates heavily enough that an ownership mistake shows up there first. the line table is now cleared in place instead of being rebuilt: handing the allocator three fresh lists per parse showed up as roughly 800 bytes a round in the growth gate.
docs/yaml.md writes out what parses, what is refused and why, the six input-driven limits, and the node-pool scope rules. examples/yaml_ops runs the same ground as examples/toml_ops so the two read side by side, and joins the native/self-host parity set.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
summary
adds
std.yaml, a yaml parser for the subset configuration files actuallyuse, built to the same shape as
std.jsonandstd.toml: a threadlocal nodepool addressed by opaque
Inthandles,type_offor the kind, the tomlaccessor names,
open_scope/close_scopewith thepool_mark_*/pool_restorepair underneath, and the full generic
decode[T]family.what parses: block mappings and sequences nested in each other, the compact
- name: aform, sequences written at their key's own indentation, flowcollections, plain scalars resolved against the yaml 1.2 core schema, single-
and double-quoted scalars with escapes including
\uXXXX, comments, literaland folded block scalars with strip and keep chomping, and multiple documents
separated by
---(parsereturns the first,parse_allreturns them all).what does not, each refused with a message that names it rather than being
quietly mis-parsed: anchors and aliases, merge keys, tags, complex keys,
directives, multi-line plain scalars, explicit block-scalar indentation
indicators, flow collections spanning lines, and content on a
---line.anchors are the interesting exclusion. alias expansion is the billion-laughs
amplification vector, and a depth cap does not help because the blowup is in
width, so doing them safely needs an expansion budget, which is a different
design. tabs in indentation are refused too; that one is yaml's own rule.
six limits bound everything the input controls: 4 MiB per document (the same
ceiling
std.net.grpcputs on a message), 64 levels of nesting, 262144 nodes,65536 entries per collection, 1 MiB per scalar and 4096 bytes per key. they are
module constants, not a table built on first use, because a lazily initialized
table is a race under the green runtime.
typed decoding needed a small compiler change.
std.tomlandstd.yamlparseinto the same thing, a node pool reached through
require_string,require_int,require_bool,has,require_tableand three pool marks, sothe emitter already knew how to lower a decode against that shape. it just
called the shape "toml". this renames the family to handle-pool decode, makes
the last three hardcoded
std_toml_names follow the call's receiver likeevery other name in that path already does, and adds
std.yamlto the set.toml's private
toml_require_tablebecomes a publicrequire_tableso bothmodules answer to one member name. the bootstrap seed is regenerated.
what was tested
make test: all fourteen steps, run in slices because the whole targetoutruns a single command window. 100 examples, 297 regression cases native
and again through the self-hosted compiler, invalid parse and checker cases,
cli regressions, the ir contract checks, 15 parity examples, docsite and
sitegen goldens.
verify-green-corpusandverify-osthread-corpus: 297 cases each, green atdefault and at one worker, and again with
PITH_GREEN=0. the 100 examplesalso run clean under
PITH_GREEN=0.make bootstrap-ir-checks-onlyandmake run-examples-self-only, since theemitter changed: ir invariants, the ir fixed point on all ten sources, and
the self-hosted example corpus.
make test-std-self-only: 76 std modules, including the five colocated testsin
std/yaml.pith.make leak-check: the newleak_yaml_parsecase grew 48kb across 200k and800k rounds, against a 2048kb limit.
make memcheck:test_yaml_structure,test_yaml_malformedandtest_yaml_derived_decodejoin the valgrind set and are clean.the malformed suite is 26 cases and each one asserts both that the parse failed
and that the message says why: unterminated quotes in a value, in a key and in
a single-quoted scalar; a bad escape and a truncated
\u; tab indentation;three kinds of indentation error; truncated flow sequences and mappings; a flow
entry with no value; trailing junk after a flow collection; each of the eight
refused features; a 400-level indentation bomb and a 512-bracket flow bomb,
both stopping at the depth cap; a 16 KiB key; and a 5 MiB document. a legal
40-level document still parses, and the parser works normally afterwards.
notes
two pre-existing leaks turned up while building the leak case. neither is
introduced here and neither is fixed here:
get_stringon a handle-pool module leaks a string per call, roughly 31bytes a round.
std.tomlmeasures identically, so it is not yaml-specific.decode_text[T]leaks roughly 280 bytes a round, again identically forstd.toml. it looks like the known result-local leak: the lowered bodystores a
parse_requiredresult in a temp and reads.okoff it.the leak case therefore exercises the scoped parse paths, which are flat.
one thing worth knowing for anyone writing a std module: assigning a fresh list
literal to a
threadlocal mut xs: List[T]leaks the old list, about 800 bytesa round in this parser's case. clearing in place avoids it, which is what
scan_linesdoes now.