diff --git a/CONFORMANCE.md b/CONFORMANCE.md index e21353d..7014d2b 100644 --- a/CONFORMANCE.md +++ b/CONFORMANCE.md @@ -53,17 +53,21 @@ files other than the document itself. | Other | `frontmatter-invalid-yaml` (reported by the CLI and assembly, not the validator), `definition-circular`, `definition-used-before-declaration`, `language-code-invalid`, `authoritative-not-declared` | In practice this means validating a set of files is out of scope: the validator does not resolve -or cross-check includes, attachment file contents, or bilingual document sets. Single-document +or cross-check includes, attachment file contents, or bilingual document sets. The one thing it +reads from another file is the definitions the amended original (§7.5) and the LegalDown +attachment files (§12.4) declare, from beside a document that has a path (`legaldown.load`; the +CLI) or through `validate(resolve=)`; an unreadable file is as if it were not asked for. Single-document authoring, editing, and CI validation are fully covered. Assembly is the exception: it reads a template's include fragments and LegalDown attachment files, and reports the checks on them that its output depends on (see [Assembly](#assembly-157-176)); the validator still does not. The template constructs of specification 0.2 (§15) are validated within the document: questions, conditions and alternatives, reference safety, `{{choose:}}`, drafting notes, insertion -boundaries, and the final option (§15.9, `validate_document(final=True)` or +boundaries, and the final option (§15.9, `validate(final=True)` or `legaldown validate --final`); assembly is described below. Two limits apply: -- A template's include fragments and LegalDown attachment files are not read (Full, §17.4), so +- A template's include fragments and LegalDown attachment files are not read (Full, §17.4) — the + attachment files only for the definitions they declare — so `question-unused` is not reported for a template that has either: a question may be used there. - A LegalDown attachment file or include fragment validated on its own is checked as a standalone document: conditions, placeholders, and terms that refer to its template's @@ -162,7 +166,7 @@ failed to resolve. From assembly, which reads the file, it means the file could and for one whose `---` block holds YAML that is a scalar or a list rather than a mapping of fields: that block is not frontmatter, its `---` lines are thematic breaks, and the whole document is validated as body. A block that cannot be read at all — YAML that is malformed, nested too -deep, or a mapping of another kind (`!!set`) — is `frontmatter-invalid-yaml`: `parse_document` +deep, or a mapping of another kind (`!!set`) — is `frontmatter-invalid-yaml`: `parse` (and `load`) raises `FrontmatterError` for it, which the CLI reports and assembly returns as a diagnostic, for the template or for a fragment or attachment file it reads. Any other exception while parsing is a fault of this implementation: the CLI reports it as an internal error (exit status 2), never as @@ -201,18 +205,21 @@ CI runs this on every push and pull request. ## Assembly (§15.7, §17.6) -`assemble(template, answers)` and `legaldown assemble` perform §15.7.2 byte for byte and report +`Template.form(answers).assemble()` and `legaldown assemble` perform §15.7.2 byte for byte and report the answer rules of §16.12 (`answer-invalid`, `answer-missing`, `answer-unknown`). The block structure assembly edits is recorded by the parser's own walk, so assembly and validation read a -template the same way. `template_questions` and `needed_questions` list the questions a template -asks, all of them or those an answers set still leaves open. +template the same way. `Template.questions` lists the questions a template asks; `Form.questions` +those it reaches given an answers set, and `Form.unanswered` those still left open. §15.7.2 gives assembly a template that +validates without Errors, which assembly itself does not check; a `Template` refuses one with Errors +in the validator's template rules (`Template.problems`), and the deprecated `assemble` function does +not. - A single-file template is assembled at Core, as §17.6 permits ("Core + Assembly"). - A template with include fragments or LegalDown attachment files is assembled when the caller - passes `load_file`, which reads them; the CLI reads them relative to the template and refuses a + passes `resolve` (`load_file` of the deprecated function), which reads them; the CLI reads them relative to the template and refuses a path that leads out of its directory. Reading them is a Full capability (§17.4) that this implementation provides for assembly without claiming Full, as §17.1 allows. Without - `load_file`, such a template is refused with `include-file-missing` or + `resolve`, such a template is refused with `include-file-missing` or `attachment-file-missing`, never assembled partially (§17.6). - A file assembly reads is checked as the Full level checks it (§16.10–§16.12), and the template is refused if it fails: frontmatter (`include-has-frontmatter`, `attachment-has-frontmatter`), a diff --git a/README.md b/README.md index df63881..e6cbfdd 100644 --- a/README.md +++ b/README.md @@ -130,6 +130,14 @@ Directories are searched recursively for `*.lgd`, `*.legaldown`, and `*.legal.md **Exit codes:** `0` clean · `1` diagnostics found (errors, or any diagnostic under `--strict`) · `2` a file could not be read. +**Changed in 0.4.0:** a document that amends another (`amends.file`, §7.5) or declares LegalDown +attachment files (§12.4) is now checked against the definitions those files declare, read from beside +it (never from outside its directory). Before, `legaldown validate` never read them, so a term only the +amended original defines was an Info (`amend-term-unresolvable`) and is now found, and an +`amend-def-override` or `def-duplicate-id` can be reported that was not. A CI check on such documents +can start to fail for that reason: it is the specification's result, and the files must be beside the +document. A file that is not there, or cannot be read, is as if it had not been asked for. + ### Assembling a template ```bash @@ -137,11 +145,49 @@ legaldown assemble template.lgd --answers answers.yaml # the document t legaldown assemble template.lgd --answers answers.yaml -o out/ # with its fragments and attachment files ``` -The answers set is a YAML mapping of question ids to answers (§15.7.1). Include fragments and +```bash +legaldown assemble template.lgd -i -o out/ # asked in the terminal for what is missing +legaldown assemble template.lgd -i --answers a.yaml --save-answers a.yaml -o out/ # ... and remembered +legaldown questions template.lgd --answers answers.yaml # what is asked, given the answers so far +legaldown questions template.lgd --answers answers.yaml --format json +``` + +`assemble -i` asks, in the terminal, for the answers `--answers` does not give, as the questions are +reached — a decision opens or closes the questions under it. Each prompt shows the question, what to +type (`yes or no`, `an amount and a currency, like 5000 EUR`) and the default in brackets, and each +answer is checked as it is typed. A decision is asked again until it is answered; an empty line leaves +a value question to its default, or its blank. The prompts go to standard error, so standard output +(or `-o`) is still the assembled template. Without a terminal on standard input, `-i` lists what must +still be answered and exits 1, or assembles if nothing must be. At the end of the input the interview +stops and assembly goes on with what was answered (the questions left are left to their defaults or +blanks, or refuse the assembly if they are decisions); Ctrl-C stops it (exit 2). A template that cannot be assembled asks nothing: its problems are +reported first. `--save-answers FILE` writes the answers used, with those typed, as YAML — also +after an interruption or a failure, so that typing is not lost — and the file is an answers set for +the next run. It is written as a whole or not at all, beside the file and moved into place, so it may +be the answers file that was read; but it is the answers, not the file's text, that are written, +so comments in it are lost (with or without `-i`). + +`questions` lists the questions reached, which are unanswered, which stop assembly, and what is wrong +with the answers; `--format json` is `Form.as_dict()` (below), for another program or an agent. It +exits 0, whatever the answers, unless the template has problems or cannot be read; `--check` makes a +form that cannot be assembled exit 1. + +The answers set is a YAML mapping of question ids to answers (§15.7.1); the shapes plain YAML gets +wrong are put right for the template's questions (`5000 EUR`, `yes`, a number as an amount). Include fragments and LegalDown attachment files are read relative to the template, never from outside its directory; the ones assembly keeps are written under `-o` at their relative paths. Answer problems are reported as `answer-invalid`, `answer-missing` and `answer-unknown` (§16.12) on stderr, and -nothing is written when there is an Error. Exit codes are as for `validate`. +nothing is written when there is an Error. A template with Errors in the validator's template rules +(questions, conditions, `{{choose:}}`, placeholders, insertions) is refused, with the rule ids, as +§15.7.2 gives assembly a valid template. Exit codes are as for `validate`. + +**Changed in 0.4.0:** `legaldown assemble` refuses a template that has Errors in the validator's +template rules (`question-invalid`, `condition-invalid`, `choose-invalid`, the `placeholder-*` rules, +`insertion-boundary`, `def-term-variable`, `drafting-note-def`, `template-fragment-invalid`, ...), +with exit 1 and the rule ids on stderr, where 0.3 assembled it. §15.7.2 gives assembly a template +that validates; a gate that assembles templates in CI can start to fail on one that never did. +`--save-answers` does not write when there was nothing to save (no answers file read, nothing typed), +so that an answers file you meant to resume from is not replaced by an empty one. ### JSON output @@ -151,7 +197,7 @@ integrations, and dashboards: ```json { "legaldown_spec": "0.2", - "validator_version": "0.3.0", + "validator_version": "0.4.0", "diagnostics": [ { "file": "contract.lgd", @@ -179,10 +225,10 @@ off. Three functions cover the common path: ```python -from legaldown import parse_document, validate_document, serialize_document +from legaldown import load, validate, serialize_document -document = parse_document(open("contract.lgd").read(), filename="contract.lgd") -result = validate_document(document) +document = load("contract.lgd") +result = validate(document) for diagnostic in result.diagnostics: print(diagnostic.level, diagnostic.rule, diagnostic.message) @@ -191,9 +237,43 @@ if result.is_valid: # no Error-level diagnostics print(serialize_document(document)) ``` -`parse_document` raises `FrontmatterError` (a `yaml.YAMLError` and a `ValueError`) when the -frontmatter cannot be read — the `frontmatter-invalid-yaml` rule, which `validate_document`, -given a parsed document, cannot report. +`load(path)` takes a `str` or `os.PathLike`, reads the file as UTF-8 and returns the `Document`. +It names the document for you: `document.filename` is the file's name and `document.path` its +absolute path (symbolic links are not followed), which is where the files it refers to are to be +looked for. For text that is not in a file (an editor buffer, an HTTP body, a test), use +`parse(text)`, which `load` is built on; it takes an optional `filename=` to name the document in +its diagnostics. + +`load` raises `FileNotFoundError` (or another `OSError`) when the file cannot be read and +`UnicodeDecodeError` when it is not UTF-8. Both it and `parse` raise `FrontmatterError` (a +`yaml.YAMLError` and a `ValueError`) when the frontmatter cannot be read — the +`frontmatter-invalid-yaml` rule, which `validate`, given a parsed document, cannot report. + +`validate(document)` checks it and returns a `ValidationResult`; `validate(document, final=True)` +is the signature-ready check (§15.9). A document that amends another (`amends.file`, §7.5) or +declares LegalDown attachment files (§12.4) is checked against the definitions those files +declare. They are read from beside the document — `document.path` is where `load` found it — and +never from outside its directory (§2.3); only the definitions each file declares itself are read, +not those it refers to in turn. A file that is not there, is not UTF-8, or has frontmatter that cannot be read is as +if it had not been asked for: no diagnostic of its own. A document from `parse(text)` has no path, +so there is nothing to read (`resolve=lambda path: None` does the same for a loaded one, to +validate it without reading anything); give `validate` a `resolve=` function, from a relative path to the +file's text (or `None`), to read them from elsewhere — a database, an upload. `file_loader(directory)` +is the same function for files on disk, and the one `parse_template` takes as `resolve=` (`LoadFile`). + +> **Deprecated:** `parse_document` is now `parse` (string; same arguments) or `load` (file), +> `validate_document` is now `validate`, and the importer callbacks `import_definitions=` and +> `import_attachment_definitions=` are replaced by `resolve=`. They still work and raise a +> `DeprecationWarning`; they are deprecated since 0.4.0 and will be removed in 0.5.0. +> +> **Changed in 0.4.0:** `ValidationResult` is a plain value of two parts, `diagnostics` and `index`. +> `errors`, `warnings` and `infos` are read-only lists taken from `diagnostics` (changing them +> changes nothing; change `diagnostics`). What validating resolved, besides what it found, moved +> to `result.index` (below): `result.sections` is now `result.index.sections`, `result.is_template` +> is `result.index.is_template`, and so on, and the five `inline_*` lists are +> `result.index.values.dates`, `.money`, `.durations`, `.fields` and `.placeholders`. The old names +> are gone. So are the result's recording methods (`error`, `warning`, `info`, `at`) and +> `used_terms`, which only the validator used. The public API is what the `legaldown` and `legaldown.validator` packages export (their `__all__`). Changes to it are listed in the notes of each @@ -205,18 +285,23 @@ change in any release; constants still only there are to be made public ### Working with the result Validating a document builds the indices the checks need — section numbers, resolved definitions, -party display text, every inline value found in the body. `ValidationResult` hands all of it back, -so a renderer or a UI can reuse the work instead of re-deriving it: +party display text, every inline value found in the body. `result.index`, a `DocumentIndex`, hands +all of it back, so a renderer or a UI can reuse the work instead of re-deriving it. The result +itself is what was found, kept nowhere but in `diagnostics`: -| Attribute | Contents | +| `result.…` | Contents | |---|---| | `diagnostics` | `Diagnostic(rule, level, message, line, file)` — the authoritative record | | `is_valid` | `True` when no Error-level diagnostic was reported | -| `errors` / `warnings` / `infos` | Message strings by severity | +| `errors` / `warnings` / `infos` | Message strings by severity, taken from `diagnostics` | | `rules(level=None)` | Set of rule ids present, optionally filtered by severity | +| `index` | A `DocumentIndex`. Values and markers in it are as written: one that is invalid is reported in `diagnostics` too, so check `is_valid` before relying on them: | + +| `result.index.…` | Contents | +|---|---| | `sections`, `section_lookup` | Numbered section index; resolves `{{ref:}}` targets. Numbers count from the shallowest heading level, and a level a heading skips counts as 1 (`#`, `###`, `##` → 1, 1.1.1, 1.2), so no two sections share a number except alternatives and what they contain (§15.8) | | `definition_lookup`, `party_lookup`, `side_lookup`, `attachment_lookup` | Resolved display text | -| `inline_dates`, `inline_money`, `inline_durations`, `inline_fields`, `inline_placeholders` | Field-spec values found in the body | +| `values` | The field-spec values the checks met, as written, as `InlineValues`: `dates`, `money`, `durations`, `fields`, `placeholders` (those of the frontmatter too; one with malformed arguments is not among them) | | `is_template` | Whether the document is a template (§15.1): it declares `questions`, carries a condition, or holds a `{{choose:}}` | | `placed_markers` | The markers in body text that apply (§5.7, §15.3), in document order: `PlacedMarker(section, block, fragment, offset, source, identifier, condition, field, item, include_only, line)` — in fragment `fragment` of `block_fragments(block)`, at `offset`, which is the block's `field` (`text`, or `suffix` after a lifted `{{ref:}}`/`{{term:}}`); `item` is the list item it marks, counted in pre-order over all the list's items, nested and empty ones included, as `list_fragments` counts them; `identifier` is `""` where it does not apply (an include-only paragraph, §12.2). Identifiers and conditions are as written: check `is_valid` before relying on them | @@ -234,13 +319,13 @@ validator reads the document with. ### Reading and editing the document model -`parse_document` returns a `Document` of plain dataclasses — `Metadata`, `Section`, `Block`, +`load` and `parse` return a `Document` of plain dataclasses — `Metadata`, `Section`, `Block`, `Side`, `Party`, `Attachment` — that you can inspect, edit, and write back out: ```python -from legaldown import parse_document, serialize_document +from legaldown import load, serialize_document -document = parse_document(source) +document = load("contract.lgd") document.metadata.governing_law = "Czech Republic" with open("contract.lgd", "w", encoding="utf-8") as handle: @@ -253,7 +338,7 @@ document's preamble (§4.4): it is unnumbered, so it lives in `document.preamble A `Section`'s `identifier` is the explicit `{#id}` written after its heading, or `""` when it has none. The identifiers the validator generates (§5.3, §5.5) are not written back into the -model: read them from `ValidationResult.sections`. +model: read them from `result.index.sections`. `document_to_dict()` / `document_from_dict()` round-trip the model through JSON-friendly structures, except the fields that describe the parsed source rather than the document, @@ -262,7 +347,7 @@ structures, except the fields that describe the parsed source rather than the do Each diagnostic names its `line` (from 1) and its `file` (the document's `filename`), as §16.9 requires: the line of the directive, marker, heading or block it is about, or of the frontmatter key — for a missing key, the key that holds it, or the frontmatter's first. Lines -come from the `source_map` that `parse_document` gives a document. A document built from a dict +come from the `source_map` that `load` and `parse` give a document. A document built from a dict has none, and one changed after parsing no longer fits its map: their diagnostics have no line (`None`) rather than a stale one. `FrontmatterError.line` is the line of YAML that cannot be read. `render_block()` renders a single block when you are driving your own layout. @@ -270,7 +355,7 @@ read. `render_block()` renders a single block when you are driving your own layo grammar — parameters in any order, quoted values decoded — and is what the validator itself uses. `definition_lookup(collect_definitions(document))` gives a document's definitions, id to term, -without validating it: the same map as `ValidationResult.definition_lookup`, except for definitions +without validating it: the same map as `result.index.definition_lookup`, except for definitions imported from an amended original or an attachment file. A term written empty (`"" {{def: x}}`) reads as its id (`id_term`), and an id that is not a valid identifier is left out. @@ -368,28 +453,137 @@ Each check reports at the severity the specification assigns it — Error, Warni ## Assembly ```python -from legaldown import assemble, needed_questions, template_questions +from legaldown import load_template + +template = load_template("nda.lgd") # reads it, and the files it includes, once +template.questions # every question, declared and implicit (§15.2) +template.problems # why it cannot be assembled, if it cannot -result = assemble(template_source, {"forum": "courts", "fee": {"amount": "5000", "currency": "EUR"}}) +form = template.form({"forum": "courts", "fee": {"amount": "5000", "currency": "EUR"}}) +form.questions # the questions reached so far, in order: ask these +form.unanswered # ... of them, with no valid answer and no default +form.blocking # ... of those, the decisions: assembly cannot run without them +form.diagnostics # what is wrong with the answers given +form.ready # assembly can run +form.complete # ... and nothing is left blank + +result = form.assemble() if result.ok: contract = result.output # the assembled template file files = result.files # assembled fragments and attachment files, by relative path for diagnostic in result.diagnostics: print(diagnostic.level, diagnostic.rule, diagnostic.message) +``` -template_questions(template_source) # every question, declared and implicit (§15.2) -needed_questions(template_source, answers) # what to ask next, given the answers so far +A template is read once; a form is a snapshot of the interview over it, so a front end asks for a +new form after every answer — a decision opens or closes the questions under it. `parse_template(text)` +is the same for source text. `load_template` raises `FrontmatterError` for frontmatter that cannot +be read, as `parse` does. + +**What a form says.** `questions` are those the assembly reaches given the answers so far: a +decision question when a condition or `{{choose:}}` using it lies in a present unit, a value +question when one of its placeholders does. A question with a default is among them (the default +answers it, and a front end can still offer it). `blocking` are the decision questions without an +answer or default, that a condition or `{{choose:}}` depends on: an unanswered value question +leaves its blank, which the specification allows, but an unanswered decision is not assembled. An answer that is not valid counts as no answer, and +does not stand in for the default: it is reported in `diagnostics` (`answer-invalid`), and so is an +answer to a question the template does not have (`answer-unknown`, a warning). `unused` lists the +questions that have an answer but are not reached; it is advice, since one answer that changes can +make many of them unused. `form.problem("fee")` says why one answer is not valid, and so does +`question.problem(answer)`, before anything is assembled. A form never changes after it is made, and +neither do the questions: a `Question` is a fixed value, a copy of the template's declaration, which +every form of the template shares (treat its `default` and `choices` as read-only). + +**Asking a person.** A question can read what a person types, so a terminal, a web form or an agent +need not know the shapes `assemble` takes (money is `{amount, currency}`, a boolean is a boolean): + +```python +answers, skipped = {}, set() +while True: + form = template.form(answers) + q = next((q for q in form.questions if answers.get(q.id) is None and q.id not in skipped), None) + if q is None: + break + try: + answer = q.from_text(input(f"{q.prompt or q.id} — enter {q.hint}: ")) + except ValueError as exc: # says what to enter + print(exc) + else: + if answer is not None: + answers[q.id] = answer + elif q in form.blocking: # a decision cannot be left open + print("An answer is needed.") + else: # the default applies, or the blank stays + skipped.add(q.id) +``` + +`question.hint` is what to type — `yes or no`, `a date, YYYY-MM-DD`, `an amount and a currency, like +5000 EUR`, `one of: courts (State courts), arbitration` — and `question.from_text(text)` turns the +text into the answer, or raises `ValueError` saying what to enter; the empty text is `None`, no +answer. What it returns always passes `question.problem`. It is strict and not locale-aware: +`yes`, `no`, `true`, `false`, `y`, `n` for a boolean; a choice by key or label, in any case (a label +two choices share is ambiguous, and a key comes before a label); `5000 EUR` for money and `30 D` +for a duration, with the amount alone where every placeholder fixes the currency or the unit; no +grouping separators, symbols or `5 000,50`. + +**Answers from a file.** `load_answers("answers.yaml")` reads the YAML mapping (`AnswersError` if it +is not one; a date such as `2026-13-45` is not YAML). Plain YAML gets some shapes wrong for a +template: `fee: 5000` is a number and not the string money wants, `forum: State courts` names a +choice by its label, a `yes` is not what a form takes. `template.coerce(answers)` puts right what it +can read without guessing — text as `question.from_text` reads it, a money amount given as a number — +and leaves the rest, a float or an unknown id, for the form to report; it never raises. A form does +not coerce by itself, so that its diagnostics stay the specification's; `legaldown assemble` and +`legaldown questions` do. + +```python +answers = template.coerce(load_answers("answers.yaml")) +form = template.form(answers) ``` +**The form as data.** `form.as_dict()` is the form as JSON-ready data for a web form, a service or an +agent: `ready`, `complete`, the template's `problems`, the `diagnostics` about the answers (each with +the `question` it is about), and `questions` — those reached first, in order, then the others — each with +`id`, `type`, `label`, `prompt`, `reached`, `blocking`, `state` (`answered`, `default`, `invalid`, +`unanswered`), `answer`, `default`, `problem`, `hint` in words and `accepts` as data: the words of a +boolean, the choices, the currency or unit the placeholders fix, the units there are. +`question.to_text(answer)` is what a person would type for an answer (`yes`, `5000 EUR`, `30 D`), +to show a default or fill in an input; `answer_text` and `default_text` hold it in the data. + +**What stops a template.** `template.problems` are the reasons a template cannot be assembled +whatever the answers: a file it includes that cannot be read, a placeholder written across lines, a +translation group, and the Errors of the validator's template rules (`question-invalid`, +`condition-invalid`, `choose-invalid`, the `placeholder-*` rules, `insertion-boundary`, +`def-term-variable`, `drafting-note-def`, `template-fragment-invalid`; §15.7.2 gives assembly a +template that validates, and `condition-reference-unsafe` counts too, for a template that includes +no fragment: the validator cannot see which sections a fragment holds), and a placeholder, outside +a drafting note, that would fill in a directive with a repeated parameter (`directive-duplicate-param`) +or a duration unit §10.5 does not define (`duration-invalid-unit`). +`template.validation` is the full `ValidationResult` of the template read alone, as advice: the +validator does not read the fragments a template includes, so it cannot see a section or a +definition that lives in one. `ready` is false while there are problems, and `form.assemble()` then +returns them as its diagnostics. + Assembly edits the template as written — "no other byte of the template changes" — so its output is identical to any other conforming implementation's. A template with include fragments -or LegalDown attachment files needs `load_file=`, a function from a relative path to the file's -text; without it such a template is refused rather than assembled partially (§17.6). The files it +or LegalDown attachment files reads them from beside the file (`load_template`) or through +`parse_template(text, resolve=...)`, a function from a relative path to the file's text; without +one such a template is refused rather than assembled partially (§17.6). The files it reads are checked as the Full level checks them — no frontmatter, no level 1 heading, and in a template no includes, and no conditions or drafting notes in a fragment — and a template whose files fail is refused. [CONFORMANCE.md](CONFORMANCE.md#assembly-157-176) lists what assembly checks and what it does not. +> **Deprecated:** `assemble(text, answers, load_file=)`, `template_questions` and `needed_questions` +> are replaced by `Template` and `Form`. They still work, unchanged (but for `Question`, below, and +> that a `load_file=` loader is never asked for a path that is absolute or leads out of the template's +> directory, §2.3), and raise a `DeprecationWarning`; they are deprecated since 0.4.0 and will be removed in 0.5.0. Two things +> differ in the new API: it asks the questions of an included fragment where its `{{include:}}` is +> (the functions ask them after the body), and it refuses a template that has Errors in the +> validator's template rules (the functions assemble it). `legaldown assemble` follows the new API, +> and puts the shapes of its answers file right (`template.coerce`): `5000 EUR` is accepted for a money +> question (and `fee: 5000`, where its placeholders fix the currency), `no` for a boolean, and surrounding spaces of a text answer are +> dropped, which were `answer-invalid` before. A `Question` is now a fixed value. + ## Scope This implementation claims **Level 1 — Core** (§17.2) and the **Assembly** capability (§17.6): diff --git a/src/legaldown/__init__.py b/src/legaldown/__init__.py index eae6f84..0d29e9c 100644 --- a/src/legaldown/__init__.py +++ b/src/legaldown/__init__.py @@ -6,10 +6,10 @@ Quick start:: - from legaldown import parse_document, serialize_document, validate_document + from legaldown import load, serialize_document, validate - doc = parse_document(open("contract.lgd").read()) - result = validate_document(doc) + doc = load("contract.lgd") + result = validate(doc) for diagnostic in result.diagnostics: print(diagnostic.level, diagnostic.rule, diagnostic.message) if result.is_valid: @@ -21,7 +21,6 @@ from .assembly import ( AssemblyError, AssemblyResult, - LoadFile, Question, assemble, needed_questions, @@ -51,6 +50,7 @@ iter_directives, lex, ) +from .files import LoadFile, file_loader from .models import ( BLOCK_DEFAULTS, Amends, @@ -77,27 +77,31 @@ ) # Parser & serializer -from .parser import FrontmatterError, collect_source_directives, parse_document +from .parser import FrontmatterError, collect_source_directives, load, parse, parse_document from .serializer import render_block, render_item, serialize_document # The LegalDown specification version this implementation targets. from .specification import SPEC_VERSION +from .template import AnswersError, Form, Template, load_answers, load_template, parse_template # Validator from .validator import ( AttachmentDefinitionsImporter, DefinitionsImporter, Diagnostic, + DocumentIndex, + InlineValues, PlacedMarker, SectionIndexEntry, ValidationResult, is_drafting_note, is_template, slugify_identifier, + validate, validate_document, ) -__version__ = "0.3.0" +__version__ = "0.4.0" #: Conformance level per specification §17. "core" — parse and validate a #: single document. Rendering and Full (multi-file: includes, attachments, @@ -106,7 +110,7 @@ CONFORMANCE_LEVEL = "core" #: Named capabilities claimed besides the level (§17.6): template assembly, -#: for single-file templates; a caller-supplied ``load_file`` also lets it +#: for single-file templates; a caller-supplied ``resolve`` also lets it #: read include fragments and LegalDown attachment files. CAPABILITIES: frozenset[str] = frozenset({"assembly"}) @@ -117,18 +121,28 @@ "CONFORMANCE_LEVEL", "CAPABILITIES", # Core workflow - "parse_document", + "load", + "parse", + "parse_document", # deprecated since 0.4.0, removed in 0.5.0: use ``parse`` or ``load`` "FrontmatterError", "serialize_document", - "validate_document", + "validate", + "validate_document", # deprecated since 0.4.0, removed in 0.5.0: use ``validate`` # Assembly (§15.7, §17.6) - "assemble", - "template_questions", - "needed_questions", + "load_template", + "parse_template", + "load_answers", + "AnswersError", + "Template", + "Form", + "assemble", # deprecated since 0.4.0, removed in 0.5.0 + "template_questions", # deprecated since 0.4.0, removed in 0.5.0 + "needed_questions", # deprecated since 0.4.0, removed in 0.5.0 "AssemblyResult", "AssemblyError", "Question", "LoadFile", + "file_loader", # Definitions (§7) "collect_definitions", "definition_lookup", @@ -151,13 +165,15 @@ "KNOWN_DIRECTIVES", "render_block", "render_item", - "AttachmentDefinitionsImporter", + "AttachmentDefinitionsImporter", # deprecated since 0.4.0, removed in 0.5.0 # Result types "ValidationResult", + "DocumentIndex", + "InlineValues", "SectionIndexEntry", "Diagnostic", "PlacedMarker", - # What validate_document decides, on its own (§15.1, §15.6) + # What validate decides, on its own (§15.1, §15.6) "is_template", "is_drafting_note", # Document model @@ -187,5 +203,5 @@ # Utilities "collect_source_directives", "slugify_identifier", - "DefinitionsImporter", + "DefinitionsImporter", # deprecated since 0.4.0, removed in 0.5.0: use ``validate(resolve=)`` ] diff --git a/src/legaldown/assembly.py b/src/legaldown/assembly.py index 0d54648..de342c8 100644 --- a/src/legaldown/assembly.py +++ b/src/legaldown/assembly.py @@ -28,16 +28,20 @@ from __future__ import annotations import bisect +import copy import posixpath import re +import reprlib +import warnings from collections.abc import Callable, Iterator, Mapping -from dataclasses import dataclass, field +from dataclasses import InitVar, dataclass, field from datetime import date from functools import cache from typing import Any from .definitions import list_fragments from .directives import PLACEHOLDER_TYPE_PARAMS, Directive, format_value, lex +from .files import LoadFile, relative_path from .markdown import ( FENCE_OPEN_RE, HTML_BLOCK_START_RE, @@ -59,13 +63,14 @@ _may_interrupt, _opens_paragraph, _row_width_end, - parse_document, + parse, quote_content, ) -from .validator import validate_document +from .validator import validate from .validator.conditions import Condition from .validator.core import is_template -from .validator.patterns import IDENTIFIER_RE, LEGALDOWN_EXTENSIONS, VALID_PLACEHOLDER_TYPES +from .validator.helpers import is_positive_numeric, is_valid_money_amount +from .validator.patterns import DURATION_UNITS, IDENTIFIER_RE, LEGALDOWN_EXTENSIONS, VALID_PLACEHOLDER_TYPES from .validator.result import Diagnostic from .validator.templates import ( DECISION_QUESTION_TYPES, @@ -77,6 +82,7 @@ from .validator.units import find_markers, is_include_only, own_presence __all__ = [ + "frontmatter_diagnostic", "AssemblyError", "AssemblyResult", "LoadFile", @@ -86,10 +92,6 @@ "template_questions", ] -#: Reads an include fragment or LegalDown attachment file by its path as the -#: template writes it (normalized); ``None`` when there is no such file. -LoadFile = Callable[[str], "str | None"] - #: A question with neither an answer nor a default. _MISSING = object() @@ -120,13 +122,18 @@ def ok(self) -> bool: return not any(d.level == "error" for d in self.diagnostics) -@dataclass(slots=True) +@dataclass(frozen=True) class Question: """One question the person filling a template is asked (§15.2): declared in ``questions``, or implicit — an undeclared placeholder, with no prompt and no default. ``type`` is the effective type; ``currency``/``unit`` is set when every placeholder for the question fixes the same one, which is - when the bare amount or value is an acceptable answer (§15.7.1).""" + when the bare amount or value is an acceptable answer (§15.7.1). + + A question is a fixed value, copied from the template's declaration: it + cannot be reassigned, and ``choices`` and a map-valued ``default`` (a money + or duration default) are copies of the declaration's. Treat them as + read-only: the question is shared by every form of its template.""" id: str type: str @@ -136,11 +143,211 @@ class Question: currency: str | None = None unit: str | None = None declared: bool = True + #: What the placeholders of the question fix (``Blank``): every code, and + #: whether one is in frontmatter, which ``problem`` needs. Not a field: it is + #: not part of the value (``asdict``, ``replace`` and equality leave it out). A + #: question made without it behaves as if its placeholders fix the ``currency`` + #: or ``unit`` it names, if any, and none is in frontmatter. + _blank: InitVar[Blank | None] = None + + def __post_init__(self, _blank: Blank | None) -> None: + object.__setattr__(self, "choices", dict(self.choices or {})) + object.__setattr__(self, "default", copy.deepcopy(self.default)) + object.__setattr__(self, "_info", _blank) @property def is_decision(self) -> bool: return self.type in DECISION_QUESTION_TYPES + def problem(self, answer: Any) -> str | None: + """Why *answer* is not a valid answer to this question (§15.7.1), or + None when it is.""" + return answer_problem( + self.type, answer, choices=self.choices if self.type == "choice" else None, blank=self._placeholders() + ) + + def _placeholders(self) -> Blank | None: + """What the placeholders fix: as recorded, else as ``currency``/``unit`` name.""" + if self._info is not None: + return self._info + fixed = self.currency if self.type == "money" else self.unit if self.type == "duration" else None + return Blank(type=self.type, codes={fixed}) if fixed else None + + @property + def _fixed(self) -> str | None: + """The currency (money) or unit (duration) every placeholder fixes, if one: + what ``problem`` goes by, so that the hint and ``from_text`` never differ from it.""" + blank = self._placeholders() + if blank is None or self.type not in ("money", "duration"): + return None + return next(iter(blank.codes)) if len(blank.codes) == 1 and "" not in blank.codes else None + + @property + def hint(self) -> str: + """What to type to answer it, as a phrase to follow "Enter": ``yes or + no``, ``an amount and a currency, like 5000 EUR``, ``one of: courts + (State courts), arbitration`` — for a person, a web form or an agent.""" + if self.type == "boolean": + return "yes or no" + if self.type == "date": + return "a date, YYYY-MM-DD" + if self.type == "choice": + if not self.choices: + return "an answer (the question declares no choices)" + return "one of: " + ", ".join( + f"{key} ({label})" if label and label.casefold() != key.casefold() else key + for key, label in self.choices.items() + ) + if self.type == "money": + if self._fixed: + return f"an amount, like 5000 (in {self._fixed})" + return "an amount and a currency, like 5000 EUR" + if self.type == "duration": + if self._fixed: + return f"a number, like 30 (in {self._fixed})" + return f"a number and a unit, like 30 D (units: {', '.join(DURATION_UNITS)})" + if self._info is not None and self._info.in_frontmatter: + return "text, without '{{'" + return "text" + + @property + def accepts(self) -> dict[str, Any]: + """What the question accepts, for a form or an agent to build its input + from: ``kind`` (the type), and what that kind needs — the words of a + boolean, the choices, the currency or unit the placeholders fix and the + units there are, whether the text fills frontmatter. ``hint`` says it in + words.""" + if self.type == "boolean": + return {"kind": "boolean", "words": {"true": ["yes", "y", "true"], "false": ["no", "n", "false"]}} + if self.type == "date": + return {"kind": "date", "format": "YYYY-MM-DD"} + if self.type == "choice": + return {"kind": "choice", "choices": [{"key": key, "label": label} for key, label in self.choices.items()]} + if self.type == "money": + return {"kind": "money", "currency": self._fixed} + if self.type == "duration": + return {"kind": "duration", "unit": self._fixed, "units": list(DURATION_UNITS)} + return {"kind": "text", "frontmatter": bool(self._info is not None and self._info.in_frontmatter)} + + def to_text(self, answer: Any) -> str: + """*answer* as a person would type it — the inverse of :meth:`from_text`: + to show a default or a saved answer, or to fill in an input. ``yes`` or + ``no`` for a boolean, ``5000 EUR`` for money, ``30 D`` for a duration, an + ISO date; the empty text for no answer. Anything else as ``str`` has it.""" + if answer is None: + return "" + if isinstance(answer, bool): + return "yes" if answer else "no" + if isinstance(answer, date): + return answer.isoformat() + if isinstance(answer, dict): + keys = ("amount", "currency") if self.type == "money" else ("value", "unit") + if self.type in ("money", "duration") and set(answer) == set(keys): + return f"{answer[keys[0]]} {answer[keys[1]]}" + try: + if isinstance(answer, (dict, list, tuple, set, frozenset)): + return _DESCRIBE.repr(answer)[:200] # not an answer it knows: described, within bounds + return str(answer) + except (ValueError, RecursionError): # an integer too long for Python to write as text + return "" + + def from_text(self, text: str) -> Any: + """The answer that *text*, as a person types it, gives to this question: + the shape ``assemble`` takes, which :meth:`problem` accepts — or ``None`` + for no answer (the empty text: the default applies, or the blank stays). + Surrounding spaces are dropped. Strict, and not locale-aware: + + - ``text``: the text. ``date``: ``YYYY-MM-DD``. + - ``boolean``: yes, no, true, false, y, n, in any case. + - ``choice``: a choice's key, else its label, in any case; a label two + choices share is ambiguous. + - ``money``: ``5000 EUR`` (or ``5000EUR``), and the amount alone where the + placeholders fix a currency. ``duration``: ``30 D``, and the number + alone where they fix a unit. No grouping separators or symbols. + + Raises ``ValueError``, saying what to enter, for text that is no answer, + and ``TypeError`` for what is not text. + """ + if not isinstance(text, str): + raise TypeError(f"the text of an answer is a string, not {type(text).__name__}") + text = text.strip() + if not text: + return None + value = self._answer_from(text) + if self.type not in ("boolean", "choice", "date", "money", "duration"): + # What a person types is held to more than the specification asks of an + # answer (§15.7.1): no key's escape sequence, no byte that is no UTF-8. + if _CONTROL_RE.search(value): + raise ValueError(f"Enter {self.hint}. (it must not contain a control character)") + try: + value.encode("utf-8") + except UnicodeEncodeError: + raise ValueError(f"Enter {self.hint}. (it must be text that can be written as UTF-8)") from None + problem = self.problem(value) + if problem: + # Said as the question is, where what is wrong is not the shape of what was typed. + detail = "" if self.type in ("boolean", "choice", "date") else f" ({problem})" + raise ValueError(f"Enter {self.hint}.{detail}") + return value + + def _answer_from(self, text: str) -> Any: + shape = ValueError(f"Enter {self.hint}.") + if self.type == "boolean": + word = text.casefold() + if word in ("yes", "y", "true"): + return True + if word in ("no", "n", "false"): + return False + raise shape + if self.type == "choice": + if text in self.choices: + return text + word = text.casefold() + keys = [key for key in self.choices if key.casefold() == word] + if keys: + return keys[0] + labels = [key for key, label in self.choices.items() if label.casefold() == word] + if len(labels) == 1: + return labels[0] + if labels: + raise ValueError(f"'{text}' is the label of more than one choice ({', '.join(labels)}); " + f"enter {self.hint}.") + raise shape + if self.type in ("money", "duration"): + found = re.fullmatch(r"(\S*?\d)\s*([A-Za-z]+)?", text) + if found is None: + raise shape + amount, code = found.group(1), (found.group(2) or "").upper() + if not (is_valid_money_amount(amount) if self.type == "money" else is_positive_numeric(amount)): + what = "an amount" if self.type == "money" else "a positive number" + shown = amount if len(amount) <= 40 else amount[:37] + "..." + raise ValueError(f"'{shown}' is not {what}: digits and at most one decimal point, with no " + f"separators or symbols. Enter {self.hint}.") + if not code: + if not self._fixed: + raise shape + return amount + names = ("amount", "currency") if self.type == "money" else ("value", "unit") + return {names[0]: amount, names[1]: code} + return text + + +#: C0 controls but the tab (line breaks are refused by ``problem``), and DEL: what a terminal's +#: keys and escape sequences say, not text. +_CONTROL_RE = re.compile(r"[\x00-\x08\x0b-\x1f\x7f]") + + +def _describer() -> reprlib.Repr: + describer = reprlib.Repr() + describer.maxlevel = 3 + describer.maxlist = describer.maxtuple = describer.maxdict = describer.maxset = describer.maxfrozenset = 4 + describer.maxstring = 40 + describer.maxother = 80 + return describer + + +#: How an answer of no known shape is described (``Question.to_text``): shallow and short. +_DESCRIBE = _describer() # ── Where the parser's blocks lie in the source ────────────────── @@ -303,7 +510,7 @@ def _read_body( ) -> _Source: lines = body.split("\n") if lines[-1] == "": - lines.pop() # the last line's ending, not a line -- as parse_document reads it + lines.pop() # the last line's ending, not a line -- as parse reads it layout = _layout(lines) found: dict[tuple[int | None, int], list] = {} for marker in find_markers(document, cache(lex)): @@ -656,12 +863,16 @@ class _Template: types: dict[str, str] # every question's effective type: declared, then implicit bom: str problems: list[Diagnostic] + document: Document + #: The template's own heading identifiers (step 7), worked out when an + #: assembly first needs them, then kept: they do not depend on the answers. + identifiers: dict | None = None def _read(template: str, load_file: LoadFile | None) -> _Template: bom = "\ufeff" if template.startswith("\ufeff") else "" source, newline = _unix(template) - document = parse_document(source) + document = parse(source) declared = document.metadata.questions if isinstance(document.metadata.questions, dict) else {} template_like = is_template(document) front, body = _split(source, document) @@ -687,7 +898,7 @@ def _read(template: str, load_file: LoadFile | None) -> _Template: types = {qid: qtype for qid in declared if (qtype := question_type(declared, qid))} for pid, blank in blanks.items(): types.setdefault(pid, blank.type or "text") - return _Template(front, main, subs, declared, blanks, types, bom, problems) + return _Template(front, main, subs, declared, blanks, types, bom, problems, document) def _read_files(main: _Source, front: _Frontmatter | None, load_file: LoadFile | None, @@ -703,7 +914,9 @@ def _read_files(main: _Source, front: _Frontmatter | None, load_file: LoadFile | for path, kind in wanted: if path in subs: continue - text = load_file(path) if load_file is not None else None + # Only a path within the template's directory (§2.3) is asked for: a loader is + # not handed ``../x``, an absolute path or a drive letter to open. + text = load_file(path) if load_file is not None and relative_path(path) is not None else None if text is None: reason = "cannot be read" if load_file else "needs the Full level (§17.6)" message = f"'{path}' {reason}; the template is not assembled." @@ -711,7 +924,7 @@ def _read_files(main: _Source, front: _Frontmatter | None, load_file: LoadFile | continue text, newline = _unix(text) try: - sub_document = parse_document(text) + sub_document = parse(text) except FrontmatterError as exc: problems.append(Diagnostic( "frontmatter-invalid-yaml", "error", @@ -837,6 +1050,7 @@ def _questions(t: _Template) -> list[Question]: currency=fixed if qtype == "money" else None, unit=fixed if qtype == "duration" else None, declared=qid in t.declared, + _blank=t.blanks.get(qid), )) return questions @@ -859,27 +1073,67 @@ def get(self, qid: str) -> Any: return _MISSING if value is None else value -def _answer_diagnostics(t: _Template, answers: Mapping[str, Any]) -> list[Diagnostic]: - diagnostics = [ - Diagnostic("answer-unknown", "warning", - f"'{key}' is neither a question nor a placeholder of the template; its answer " - f"is ignored (§15.7.1).") - for key in answers - if not isinstance(key, str) or (key not in t.types and key not in t.declared) - ] +def _answer_problem(t: _Template, qid: str, answer: Any) -> str | None: + """Why *answer* is not a valid answer to question *qid* of *t*, or None.""" + qtype = t.types.get(qid) + if qtype is None: + return None + declaration = t.declared.get(qid) + choices = declaration.get("choices") if isinstance(declaration, dict) else None + return answer_problem(qtype, answer, choices=choices, blank=t.blanks.get(qid)) + + +def _answer_pairs(t: _Template, answers: Mapping[str, Any]) -> Iterator[tuple[str, Diagnostic]]: + """What is wrong with *answers*, as ``(question id, diagnostic)``: a key + that is no question (its own id), and every answer that is not valid (the + answer to a question not reached included).""" + for key in answers: + if not isinstance(key, str) or (key not in t.types and key not in t.declared): + yield str(key), Diagnostic( + "answer-unknown", "warning", + f"'{key}' is neither a question nor a placeholder of the template; its answer " + f"is ignored (§15.7.1).") resolved = _Answers(answers, t.declared) - for qid, qtype in t.types.items(): + for qid in t.types: answer = resolved.get(qid) if answer is _MISSING: continue - declaration = t.declared.get(qid) - choices = declaration.get("choices") if isinstance(declaration, dict) else None - problem = answer_problem(qtype, answer, choices=choices, blank=t.blanks.get(qid)) + problem = _answer_problem(t, qid, answer) if problem: whose = "answer to" if answers.get(qid) is not None else "default of" - diagnostics.append(Diagnostic("answer-invalid", "error", - f"The {whose} '{qid}' {problem} (§15.7.1).")) - return diagnostics + yield qid, Diagnostic("answer-invalid", "error", + f"The {whose} '{qid}' {problem} (§15.7.1).") + + +def _missing_diagnostic(qid: str) -> Diagnostic: + return Diagnostic("answer-missing", "error", + f"'{qid}' decides what the assembled document holds, and has neither an " + f"answer nor a default (§15.7.2 step 1).") + + +class _FormAnswers(_Answers): + """The answers of a form: as ``_Answers``, except that an answer that is + not valid counts as no answer — it does not steer which units are present, + and the default does not stand in for it. It is reported on its own + (``answer-invalid``).""" + + def __init__(self, answers: Mapping[str, Any], t: _Template) -> None: + super().__init__(answers, t.declared) + self._t = t + self._valid: dict[str, bool] = {} + + def get(self, qid: str) -> Any: + explicit = self._answers.get(qid) + if explicit is not None: + if qid not in self._valid: + self._valid[qid] = _answer_problem(self._t, qid, explicit) is None + if not self._valid[qid]: + return _MISSING + return super().get(qid) + + +def _answer_diagnostics(t: _Template, answers: Mapping[str, Any]) -> list[Diagnostic]: + return [diagnostic for _qid, diagnostic in _answer_pairs(t, answers)] @dataclass(slots=True) @@ -893,10 +1147,14 @@ class _Decision: missing: dict[str, None] = field(default_factory=dict) -def _decide(t: _Template, answers: _Answers) -> _Decision: +def _decide(t: _Template, answers: _Answers, *, inline: bool = False) -> _Decision: """Evaluate every condition that is reached (§15.7.2 steps 1–2). A unit inside a removed one is never reached, so its question is not needed; one - under an unanswered condition is not reached *yet*.""" + under an unanswered condition is not reached *yet*. With *inline*, an + include fragment is read where its ``{{include:}}`` is, so that the + questions are in the order they are reached; otherwise after the body. + Which lines are removed, and which files, is the same either way (a file + included twice is read at its last include, which decides it).""" decision = _Decision() def holds(test: Condition | None) -> bool | None: @@ -928,7 +1186,18 @@ def use(occ: _Occurrence) -> None: for occ in front.occurrences if front else []: if occ.line not in dropped: # a blank in a removed attachment's title use(occ) - decision.removed[""], pending = _walk(t.main, holds, use) + walked: set[str] = set() + + # A file included twice is kept or removed as its last include says. + last = {include.path: include for include in t.main.includes} + + def enter(include: _Include) -> None: + source = t.subs.get(include.path) + if source is not None and last[include.path] is include and include.path not in walked: + walked.add(include.path) + decision.removed[include.path], _pending = _walk(source, holds, use) + + decision.removed[""], pending = _walk(t.main, holds, use, enter if inline else None) for include in t.main.includes: if include.start in decision.removed[""]: decision.files[include.path] = False @@ -938,24 +1207,31 @@ def use(occ: _Occurrence) -> None: if item.file: decision.files[item.file] = kept for path, source in t.subs.items(): - if decision.files.get(path): + if decision.files.get(path) and path not in walked: decision.removed[path], _pending = _walk(source, holds, use) return decision -def _walk(source: _Source, holds: Callable, use: Callable) -> tuple[set[int], set[int]]: +def _walk( + source: _Source, holds: Callable, use: Callable, enter: Callable | None = None +) -> tuple[set[int], set[int]]: """The lines of *source* assembly removes — drafting notes first, then absent units — and those waiting on an unanswered question. Units and insertions are visited in source order, so an enclosing unit is decided - before anything inside it.""" + before anything inside it. *enter*, when given, is called with each include + that is present, after the units that start on its line.""" removed, pending = set(source.notes), set() events = sorted( [(unit.start, 0, k) for k, unit in enumerate(source.units)] + [(occ.line, 1, k) for k, occ in enumerate(source.occurrences)] + + ([(include.start, 2, k) for k, include in enumerate(source.includes)] if enter else []) ) for line, kind, k in events: if line in removed or line in pending: continue + if kind == 2: + enter(source.includes[k]) + continue if kind == 1: use(source.occurrences[k]) continue @@ -1392,8 +1668,8 @@ def _identifiers(head: str, rows: list[_Row]) -> dict[_Key, tuple[str, bool]]: """Each heading's identifier in the combined document, and whether it is explicit — generated by the validator, which knows alternatives (§5.5).""" lines = [text for text, _key in rows] - document = parse_document(head + "\n".join(lines)) - entries = validate_document(document).sections + document = parse(head + "\n".join(lines)) + entries = validate(document).index.sections headings = _layout(lines).headings if len(headings) != len(entries): raise AssemblyError("The combined document's headings do not match its parsed sections.") @@ -1405,6 +1681,12 @@ def _identifiers(head: str, rows: list[_Row]) -> dict[_Key, tuple[str, bool]]: def _template_identifiers(t: _Template) -> dict[_Key, tuple[str, bool]]: + if t.identifiers is None: + t.identifiers = _compute_template_identifiers(t) + return t.identifiers + + +def _compute_template_identifiers(t: _Template) -> dict[_Key, tuple[str, bool]]: front = t.front files = [(item.file, t.subs[item.file].lines, str(item.test) if item.test else "") for item in (front.items if front else []) if item.file in t.subs] @@ -1492,28 +1774,45 @@ def assemble( ) -> AssemblyResult: """Assemble *template* with *answers* (§15.7.2). + Deprecated since 0.4.0, removed in 0.5.0: ``parse_template(text, + resolve=...).form(answers).assemble()``, which reads the template once. + A draft or an ordinary document goes through the same steps: its drafting notes are removed, its answered blanks filled and its blank lines collapsed. *load_file* reads the include fragments and LegalDown attachment files; the assembled ones are returned in ``files``. """ + warnings.warn( + "legaldown.assemble() is deprecated since 0.4.0 and will be removed in 0.5.0; " + "use legaldown.load_template() or parse_template(), then .form(answers).assemble()", + DeprecationWarning, + stacklevel=2, + ) try: t = _read(template, load_file) except FrontmatterError as exc: - return AssemblyResult(diagnostics=[Diagnostic( - "frontmatter-invalid-yaml", "error", - f"The template's frontmatter cannot be read, so it is not assembled: {exc}", - )]) + return AssemblyResult(diagnostics=[frontmatter_diagnostic(exc)]) if t.problems: - return AssemblyResult(diagnostics=t.problems) + return AssemblyResult(diagnostics=list(t.problems)) resolved = _Answers(answers, t.declared) decision = _decide(t, resolved) - diagnostics = _answer_diagnostics(t, answers) + [ - Diagnostic("answer-missing", "error", - f"'{qid}' decides what the assembled document holds, and has neither an " - f"answer nor a default (§15.7.2 step 1).") - for qid in decision.missing - ] + diagnostics = _answer_diagnostics(t, answers) + [_missing_diagnostic(qid) for qid in decision.missing] + return _emit_result(t, resolved, decision, diagnostics) + + +def frontmatter_diagnostic(exc: FrontmatterError) -> Diagnostic: + """The error assembly reports for frontmatter that cannot be read.""" + return Diagnostic( + "frontmatter-invalid-yaml", "error", + f"The template's frontmatter cannot be read, so it is not assembled: {exc}", + ) + + +def _emit_result( + t: _Template, resolved: _Answers, decision: _Decision, diagnostics: list[Diagnostic] +) -> AssemblyResult: + """Steps 3–8 on a template whose answers have been decided; nothing is + emitted while *diagnostics* hold an Error (§15.7.2 step 1).""" if any(d.level == "error" for d in diagnostics): return AssemblyResult(diagnostics=diagnostics) head = _emit_frontmatter(t.front, decision, resolved) if t.front is not None else "" @@ -1540,7 +1839,15 @@ def template_questions( """Every question of *template* (§15.2): the declared ones in declaration order, then each undeclared placeholder, in the order first written. Raises ``FrontmatterError`` when the template's frontmatter cannot be - read.""" + read. + + Deprecated since 0.4.0, removed in 0.5.0: ``parse_template(...).questions``.""" + warnings.warn( + "legaldown.template_questions() is deprecated since 0.4.0 and will be removed in 0.5.0; " + "use legaldown.load_template() or parse_template(), then .questions", + DeprecationWarning, + stacklevel=2, + ) return _questions(_read(template, load_file)) @@ -1556,7 +1863,15 @@ def needed_questions( a value question when one of its placeholders does. A unit under a condition whose question is still unanswered is not reached yet, so the form grows as decisions are made. Raises ``FrontmatterError`` when the - template's frontmatter cannot be read.""" + template's frontmatter cannot be read. + + Deprecated since 0.4.0, removed in 0.5.0: ``parse_template(...).form(answers).questions``.""" + warnings.warn( + "legaldown.needed_questions() is deprecated since 0.4.0 and will be removed in 0.5.0; " + "use legaldown.load_template() or parse_template(), then .form(answers).questions", + DeprecationWarning, + stacklevel=2, + ) t = _read(template, load_file) decision = _decide(t, _Answers(answers, t.declared)) by_id = {question.id: question for question in _questions(t)} diff --git a/src/legaldown/cli.py b/src/legaldown/cli.py index ae0b810..5aa1bec 100644 --- a/src/legaldown/cli.py +++ b/src/legaldown/cli.py @@ -15,18 +15,23 @@ from __future__ import annotations import argparse +import contextlib import json -import posixpath +import os +import stat import sys +import tempfile from pathlib import Path from typing import Any import yaml from . import SPEC_VERSION, __version__ -from .assembly import assemble -from .parser import FrontmatterError, parse_document -from .validator import validate_document +from .assembly import AssemblyResult, frontmatter_diagnostic +from .files import within +from .parser import FrontmatterError, load +from .template import AnswersError, Form, Template, load_answers, load_template +from .validator import validate # Exit codes: 0 clean, 1 diagnostics found, 2 usage/IO failure. EXIT_OK = 0 @@ -39,12 +44,9 @@ def _validate_path(path: Path, *, final: bool = False) -> tuple[list[dict], str | None]: """Validate one file; return (diagnostics, read/parse failure message).""" try: - source = path.read_text(encoding="utf-8") - except OSError as exc: + document = load(path) + except (OSError, UnicodeDecodeError) as exc: return [], f"cannot read {path}: {exc}" - - try: - document = parse_document(source, filename=path.name) except FrontmatterError as exc: return [ { @@ -63,7 +65,14 @@ def _validate_path(path: Path, *, final: bool = False) -> tuple[list[dict], str f"(please report this)" ) - result = validate_document(document, final=final) + try: + result = validate(document, final=final) + except Exception as exc: + # As above: validating reads the files the document refers to, too. + return [], ( + f"internal error while validating {path}: {type(exc).__name__}: {exc} " + f"(please report this)" + ) return [ { "file": str(path), @@ -153,68 +162,223 @@ def _read(path: Path) -> str: return handle.read() +def _write_stdout(text: str, errors: str = "strict") -> None: + """*text* to standard output as UTF-8 bytes, whatever the terminal's encoding + and line-break translation. A character UTF-8 cannot hold (a lone surrogate) is + an error, unless *errors* says to write it escaped.""" + data = text.encode("utf-8", errors=errors) # before anything is written + sys.stdout.flush() + stream = getattr(sys.stdout, "buffer", None) + if stream is not None: + stream.write(data) + stream.flush() + else: + sys.stdout.write(text) + + +def _write_atomically(path: Path, text: str) -> None: + """*text* to *path*: all of it, or the file as it was — written beside it and + moved into place, since the file may be the answers a person typed. A symbolic + link is written through, and the file keeps its permissions (a new one gets + those a file made now would have).""" + target = path.resolve() + target.parent.mkdir(parents=True, exist_ok=True) + if target.exists(): + mode = stat.S_IMODE(target.stat().st_mode) + else: + mask = os.umask(0) + os.umask(mask) + mode = 0o666 & ~mask + descriptor, name = tempfile.mkstemp(dir=target.parent, prefix=f".{target.name}.", suffix=".tmp") + try: + with os.fdopen(descriptor, "w", encoding="utf-8", newline="") as handle: + handle.write(text) + os.chmod(name, mode) + os.replace(name, target) + except BaseException: + with contextlib.suppress(OSError): + os.unlink(name) + raise + + def _write(path: Path, text: str) -> None: path.parent.mkdir(parents=True, exist_ok=True) with path.open("w", encoding="utf-8", newline="") as handle: handle.write(text) -def _within(base: Path, relative: str) -> Path | None: - """*relative* under directory *base*, or None when it is absolute, leads - out of it (§2.3), or is no path at all (a null byte, a symlink loop).""" - if not relative or posixpath.isabs(relative) or Path(relative).is_absolute(): - return None - try: - root = base.resolve() - target = (root / relative).resolve() - except (OSError, ValueError, RuntimeError): - return None - return target if target.is_relative_to(root) else None - - def _read_answers(path: Path | None) -> tuple[dict[str, Any] | None, str | None]: """The answers set (§15.7.1): a YAML mapping, or empty without a file.""" if path is None: return {}, None try: - answers = yaml.safe_load(_read(path)) + return load_answers(path), None except OSError as exc: return None, f"cannot read {path}: {exc}" - except (yaml.YAMLError, ValueError) as exc: # a date such as 2026-13-45 raises ValueError - return None, f"cannot read the answers in {path}: {exc}" - if answers is None: - return {}, None - if not isinstance(answers, dict): - return None, f"the answers in {path} must be a YAML mapping of question ids to answers" - return answers, None + except AnswersError as exc: + return None, str(exc) + + +def _ask(prompt: str) -> str | None: + """A line from standard input, after *prompt* on standard error (standard output + is the assembled template); ``None`` at the end of the input. A line that is not + text in the terminal's encoding is asked for again. It reads the line as it is: + no editing or history, which Python gives only where standard output is the + terminal.""" + while True: + print(prompt, end="", file=sys.stderr, flush=True) + try: + line = sys.stdin.readline() + except UnicodeDecodeError: + print("That is not text in this terminal's encoding; type it again.", file=sys.stderr, flush=True) + continue + return line.rstrip("\r\n") if line else None + + +def _say(message: str) -> None: + print(message, file=sys.stderr, flush=True) + + +def _interview(template: Template, answers: dict, ask=_ask, say=_say) -> bool: + """Ask for the answers that *answers* does not give, as the form grows, until + the form has nothing left to ask; *answers* gets what is typed. A decision + cannot be left open, so it is asked again until it is answered; the empty + line leaves a value question to its default, or its blank. Returns False at + the end of the input.""" + skipped: set[str] = set() + while True: + form = template.form(answers) + question = next( + (q for q in form.questions + if (answers.get(q.id) is None or form.problem(q.id)) and q.id not in skipped), + None, + ) + if question is None: + return True + given = answers.get(question.id) + if given is not None: + say(f"The answer given to '{question.id}', {question.to_text(given)!r}, is not valid: " + f"{form.problem(question.id)}") + default = f" [{question.to_text(question.default)}]" if question.default is not None else "" + line = ask(f"{question.prompt or question.id} ({question.hint}){default}: ") + if line is None: + return False + try: + answer = question.from_text(line) + except ValueError as exc: + say(str(exc)) + continue + if answer is not None: + answers[question.id] = answer + elif question in form.blocking: + say("An answer is needed.") + else: + answers.pop(question.id, None) # an answer that is not valid does not stay: the default, or the blank + if question.default is not None: + say(f" {question.id}: the default, {question.to_text(question.default)}") + skipped.add(question.id) + + +def _missing_report(template: Template, answers: dict) -> str | None: + """What must still be answered before assembly, as text; None when nothing.""" + form = template.form(answers) + if not form.blocking: + return None + lines = [f"error: {len(form.blocking)} question(s) must be answered before assembly:"] + lines += [f" {q.id}: {q.prompt or q.id} (enter {q.hint})" for q in form.blocking] + lines.append("run in a terminal with -i to be asked, or give the answers with --answers") + return "\n".join(lines) + + +def _save_answers(path: Path, answers: dict) -> str | None: + """*answers* to the YAML file at *path*; a failure as text. Money amounts stay + quoted strings, dates the dates they are.""" + try: + text = yaml.safe_dump(answers, sort_keys=False, allow_unicode=True, default_flow_style=False) + text.encode("utf-8") + _write_atomically(path, text) + except (OSError, ValueError, RuntimeError, yaml.YAMLError) as exc: # a UnicodeEncodeError is a ValueError; a link loop a RuntimeError + return f"cannot write the answers to {path}: {exc}" + return None + + +def _same_file(first: Path, second: Path) -> bool: + """Whether the two paths are one file: the same path, or a link, symbolic or hard, to it.""" + try: + if first.resolve() == second.resolve(): + return True + return os.path.samefile(first, second) + except (OSError, RuntimeError): # one of them is not there, or a symbolic link loops + return False def _run_assemble(args: argparse.Namespace) -> int: + state: dict[str, Any] = {"answers": None} + code = EXIT_ERROR + try: + code = _assemble(args, state) + except KeyboardInterrupt: + print("\ninterrupted", file=sys.stderr) + finally: + # Also after an interruption or a failure: what was typed is not lost. + if args.save_answers and state["answers"] is not None: + if not state["answers"] and not args.answers: + # Nothing was read and nothing typed: an existing file is not replaced by an empty one. + print(f"no answers to save; {args.save_answers} was not written", file=sys.stderr) + else: + problem = _save_answers(Path(args.save_answers), state["answers"]) + if problem: + print(f"error: {problem}", file=sys.stderr) + code = EXIT_ERROR if code == EXIT_OK else code + return code + + +def _assemble(args: argparse.Namespace, state: dict[str, Any]) -> int: template_path = Path(args.template) + if args.save_answers and _same_file(Path(args.save_answers), template_path): + print("error: --save-answers names the template itself; it would be overwritten", file=sys.stderr) + return EXIT_ERROR answers, failure = _read_answers(Path(args.answers) if args.answers else None) + state["answers"] = answers + result = None if failure is None: try: - template = _read(template_path) + template = load_template(template_path) + except FrontmatterError as exc: + result = AssemblyResult(diagnostics=[frontmatter_diagnostic(exc)]) except (OSError, UnicodeDecodeError) as exc: failure = f"cannot read {template_path}: {exc}" + except Exception as exc: + # Not the template's fault: a bug in this validator, as in ``validate``. + failure = f"internal error while reading {template_path}: {type(exc).__name__}: {exc} (please report this)" if failure is not None: print(f"error: {failure}", file=sys.stderr) return EXIT_ERROR - base = template_path.parent - - def load_file(relative: str) -> str | None: - target = _within(base, relative) + if result is None: try: - return _read(target) if target is not None and target.is_file() else None - except (OSError, ValueError): # UnicodeDecodeError is a ValueError - return None - - result = assemble(template, answers, load_file=load_file) + answers = state["answers"] = template.coerce(answers) + if args.interactive and not template.problems: # a template that cannot be assembled asks nothing + if sys.stdin.isatty(): + if not _interview(template, answers): + _say("End of the input: the questions left are left to their defaults, or their blanks.") + else: + report = _missing_report(template, answers) + if report: + print(report, file=sys.stderr) + return EXIT_DIAGNOSTICS + result = template.form(answers).assemble() + except Exception as exc: + # Not the template's fault: a bug in this validator, as in ``validate``. + print(f"error: internal error while assembling {template_path}: {type(exc).__name__}: {exc} " + f"(please report this)", file=sys.stderr) + return EXIT_ERROR for d in result.diagnostics: where = f"{template_path}:{d.line}" if d.line else f"{template_path}" print(f"{where}: {d.level}: [{d.rule}] {d.message}", file=sys.stderr) if not result.ok: + if not args.interactive and any(d.rule == "answer-missing" for d in result.diagnostics): + print("run with -i in a terminal to be asked for them, or give them with --answers", file=sys.stderr) return EXIT_DIAGNOSTICS if args.output is None: @@ -226,13 +390,12 @@ def load_file(relative: str) -> str | None: ) return EXIT_ERROR # Bytes, so that no platform translates the template's line breaks. - sys.stdout.flush() - stream = getattr(sys.stdout, "buffer", None) - if stream is not None: - stream.write(result.output.encode("utf-8")) - stream.flush() - else: - sys.stdout.write(result.output) + try: + _write_stdout(result.output) + except UnicodeEncodeError as exc: + print(f"error: the assembled template holds text that cannot be written as UTF-8 ({exc.reason}); " + f"it was not written", file=sys.stderr) + return EXIT_ERROR return EXIT_OK out = Path(args.output) @@ -242,7 +405,7 @@ def load_file(relative: str) -> str | None: elif template_path.name in result.files: problem = f"{template_path.name} is both the template and a file it keeps" outputs = {template_path.name: result.output, **result.files} - targets = {relative: _within(out, relative) for relative in outputs} + targets = {relative: within(out, relative) for relative in outputs} for relative, target in targets.items(): if problem is None and target is None: problem = f"{relative} leads out of the output directory" @@ -254,12 +417,90 @@ def load_file(relative: str) -> str | None: for relative, text in outputs.items(): try: _write(targets[relative], text) # an emptied file is written as zero bytes - except OSError as exc: + except (OSError, UnicodeEncodeError) as exc: print(f"error: cannot write {relative}: {exc}; the output is incomplete", file=sys.stderr) return EXIT_ERROR return EXIT_OK +def _load_for_questions(args: argparse.Namespace) -> tuple[Template | None, dict | None, list[dict], str | None]: + """The template and the answers for ``questions``: ``(template, answers, problems, failure)``.""" + answers, failure = _read_answers(Path(args.answers) if args.answers else None) + if failure is not None: + return None, None, [], failure + path = Path(args.template) + try: + return load_template(path), answers, [], None + except FrontmatterError as exc: + diagnostic = frontmatter_diagnostic(exc) + return None, None, [{"rule": diagnostic.rule, "level": diagnostic.level, + "message": diagnostic.message, "line": diagnostic.line}], None + except (OSError, UnicodeDecodeError) as exc: + return None, None, [], f"cannot read {path}: {exc}" + except Exception as exc: + return None, None, [], f"internal error while reading {path}: {type(exc).__name__}: {exc} (please report this)" + + +def _format_form(form: Form) -> str: + """The form as text: the questions reached, then what is wrong, then whether it is ready.""" + data = form.as_dict() + lines = [] + for question in data["questions"]: + if not question["reached"]: + continue + mark = "!" if question["blocking"] else "?" if question["state"] in ("unanswered", "invalid") else " " + if question["state"] == "answered": + detail = f"answered: {question['answer_text']}" + elif question["state"] == "default": + detail = f"default: {question['default_text']}" + elif question["state"] == "invalid": + detail = f"invalid: {question['problem']}" + else: + detail = f"enter {question['hint']}" + lines.append(f"{mark} {question['id']} ({question['type']}) {question['label']} — {detail}") + others = sum(1 for question in data["questions"] if not question["reached"]) + if others: + lines.append(f" ({others} more, not asked yet or not at all with these answers)") + for d in data["problems"]: + lines.append(f"problem: [{d['rule']}] {d['message']}") + for d in data["diagnostics"]: + lines.append(f"{d['level']}: [{d['rule']}] {d['message']}") + lines.append("complete" if data["complete"] else "ready, with blanks left" if data["ready"] else "not ready") + return "\n".join(lines) + "\n" + + +def _run_questions(args: argparse.Namespace) -> int: + template, answers, problems, failure = _load_for_questions(args) + if failure is not None: + print(f"error: {failure}", file=sys.stderr) + return EXIT_ERROR + if template is None: # frontmatter that cannot be read + if args.format == "json": + _write_stdout(json.dumps({"ready": False, "complete": False, "problems": problems, + "diagnostics": [], "questions": []}, indent=2, allow_nan=False) + "\n", + errors="backslashreplace") + else: + for d in problems: + print(f"{args.template}: {d['level']}: [{d['rule']}] {d['message']}", file=sys.stderr) + return EXIT_DIAGNOSTICS + try: + form = template.form(template.coerce(answers)) + if args.format == "json": + text = json.dumps(form.as_dict(), indent=2, ensure_ascii=False, allow_nan=False) + "\n" + else: + text = _format_form(form) + ready, problems_found = form.ready, bool(template.problems) + except Exception as exc: + # Not the template's fault: a bug in this validator, as in ``validate``. + print(f"error: internal error while listing the questions of {args.template}: " + f"{type(exc).__name__}: {exc} (please report this)", file=sys.stderr) + return EXIT_ERROR + _write_stdout(text, errors="backslashreplace") + if problems_found or (args.check and not ready): + return EXIT_DIAGNOSTICS + return EXIT_OK + + def build_parser() -> argparse.ArgumentParser: parser = argparse.ArgumentParser( prog="legaldown", @@ -336,7 +577,44 @@ def build_parser() -> argparse.ArgumentParser: "written to standard output." ), ) + assemble_cmd.add_argument( + "-i", + "--interactive", + action="store_true", + help=( + "Ask, in the terminal, for the answers --answers does not give, as the " + "questions are reached (prompts go to standard error). Without a terminal " + "it lists what must still be answered and exits 1." + ), + ) + assemble_cmd.add_argument( + "--save-answers", + metavar="FILE", + help="Write the answers used, with those typed, to FILE (YAML), also after an interruption.", + ) assemble_cmd.set_defaults(func=_run_assemble) + + questions_cmd = sub.add_parser( + "questions", + help="List what a template asks, given the answers so far (§15.7).", + description=( + "List the questions a template asks given the answers so far: those reached, " + "which are unanswered, which stop assembly, and what is wrong with the answers. " + "Exit status 0 unless the template has problems or cannot be read, or, with " + "--check, is not ready to assemble." + ), + ) + questions_cmd.add_argument("template", help="The template file.") + questions_cmd.add_argument( + "--answers", metavar="FILE", help="YAML mapping of question ids to answers (§15.7.1)." + ) + questions_cmd.add_argument( + "--format", choices=("text", "json"), default="text", help="Output format (default: text)." + ) + questions_cmd.add_argument( + "--check", action="store_true", help="Exit non-zero when assembly cannot run with these answers." + ) + questions_cmd.set_defaults(func=_run_questions) return parser diff --git a/src/legaldown/definitions.py b/src/legaldown/definitions.py index 46084a8..caa8607 100644 --- a/src/legaldown/definitions.py +++ b/src/legaldown/definitions.py @@ -403,7 +403,7 @@ def id_term(def_id: str) -> str: def definition_lookup(refs: list[DefinitionRef]) -> dict[str, str]: """Build an ``{id: term}`` lookup from definition refs (first wins), as - ``validate_document`` builds its ``definition_lookup``: an empty term + ``validate`` builds its ``index.definition_lookup``: an empty term reads as its id (``id_term``), and an id that is not a valid identifier (anchor-format) is not registered.""" lookup: dict[str, str] = {} diff --git a/src/legaldown/files.py b/src/legaldown/files.py new file mode 100644 index 0000000..dcb8fa6 --- /dev/null +++ b/src/legaldown/files.py @@ -0,0 +1,63 @@ +"""Reading the files a document refers to. + +A document names other files by a path relative to itself: the document it +amends (§7.5), its include fragments (§12.2), its LegalDown attachment files +(§12.4). ``LoadFile`` is how a caller answers for such a path; +``file_loader`` is the answer for files on disk. +""" +from __future__ import annotations + +import os +import posixpath +from collections.abc import Callable +from pathlib import Path + +#: Reads a file the document refers to, by its path as the document writes it +#: (relative, ``/``-separated); ``None`` when there is no such file. +LoadFile = Callable[[str], "str | None"] + + +def relative_path(path: str) -> str | None: + """*path*, as a document names a file it refers to, normalized — ``/``-separated, + within the directory of the document (§2.3) — or None when it is not such a path: + empty, absolute, with a drive letter or a backslash or a null byte, or leading out + of the directory. A ``LoadFile`` is asked for these paths and no others.""" + if not path or "\\" in path or "\0" in path or posixpath.isabs(path) or (len(path) > 1 and path[0].isalpha() and path[1] == ":"): + return None + normalized = posixpath.normpath(path) + if normalized in (".", "..") or normalized.startswith("../"): + return None + return normalized + + +def file_loader(base: str | os.PathLike[str]) -> LoadFile: + """A ``LoadFile`` reading from directory *base* on disk. + + A path that is absolute, leads out of *base* (§2.3), is no path at all (a + null byte, a symbolic-link loop), or names no UTF-8 file reads as ``None``. + The text is returned as written, with no line-break translation, since + assembly edits it byte for byte (§15.7.2). + """ + root = Path(base) + + def load_file(relative: str) -> str | None: + target = within(root, relative) + try: + return target.read_bytes().decode("utf-8") if target is not None and target.is_file() else None + except (OSError, ValueError): # UnicodeDecodeError is a ValueError + return None + + return load_file + + +def within(base: Path, relative: str) -> Path | None: + """*relative* under directory *base*, or None when it is absolute, leads + out of it, or is no path at all.""" + if not relative or posixpath.isabs(relative) or Path(relative).is_absolute(): + return None + try: + root = base.resolve() + target = (root / relative).resolve() + except (OSError, ValueError, RuntimeError): + return None + return target if target.is_relative_to(root) else None diff --git a/src/legaldown/models.py b/src/legaldown/models.py index 18ed8df..4ad60af 100644 --- a/src/legaldown/models.py +++ b/src/legaldown/models.py @@ -7,6 +7,7 @@ from collections.abc import Iterator from dataclasses import asdict, dataclass, field +from pathlib import Path from typing import Any from .markdown import LINE_ENDING_RE, is_blank, strip_text @@ -105,12 +106,12 @@ class Metadata: #: Frontmatter keys (``questions``, ``attachments``) the parsed source #: does not write as §15.2 requires for assembly to edit them line by #: line: in YAML block style, each attachment entry beginning with - #: ``id``. Only ``parse_document`` sets it: a document built or rebuilt + #: ``id``. Only ``parse`` and ``load`` set it: a document built or rebuilt #: from a dict has no source, and the serializer writes block style. not_line_editable: list[str] = field(default_factory=list) #: True when the parsed source has no frontmatter (§3.1): no ``---`` #: block opens it, or the block's YAML is not a mapping of fields, in - #: which case the source is all body. Only ``parse_document`` sets it, + #: which case the source is all body. Only ``parse`` and ``load`` set it, #: like ``not_line_editable``. frontmatter_absent: bool = False @@ -190,11 +191,16 @@ class Document: filename: str = "" preamble: list[Block] = field(default_factory=list) #: Where the parsed source holds its parts (``positions.SourceMap``), for - #: diagnostics that name their line (§16.9). Only ``parse_document`` sets - #: it; it describes the source as parsed, so a document changed afterwards + #: diagnostics that name their line (§16.9). Only ``parse`` and ``load`` + #: set it; it describes the source as parsed, so a document changed afterwards #: may no longer fit it (then its diagnostics name no line). Not part of #: equality or of ``document_to_dict``. source_map: Any = field(default=None, compare=False, repr=False) + #: The file the document was loaded from, absolute (``load`` sets it), for + #: resolving the files it refers to. ``None`` for a document + #: parsed from a string or built in code. Not part of equality or of + #: ``document_to_dict``: where a document lives is not what it says. + path: Path | None = field(default=None, compare=False) def iter_indexed_blocks(self) -> Iterator[tuple[int | None, int, Block]]: """Every body block in document order, as ``(section_index, index, diff --git a/src/legaldown/parser.py b/src/legaldown/parser.py index b653fe6..4518e0c 100644 --- a/src/legaldown/parser.py +++ b/src/legaldown/parser.py @@ -9,10 +9,13 @@ from __future__ import annotations import bisect +import os import re +import warnings from collections.abc import Iterator from dataclasses import asdict, dataclass, field from functools import lru_cache +from pathlib import Path from typing import Any import yaml @@ -152,7 +155,7 @@ class FrontmatterError(yaml.YAMLError, ValueError): """The frontmatter cannot be read (§3.1, frontmatter-invalid-yaml): its YAML is malformed, nested too deep, or a mapping of another kind (a ``!!set``). A ``yaml.YAMLError`` and a ``ValueError``, as what - ``parse_document`` raised for it before. ``line`` is the file line (from + ``parse`` raised for it before. ``line`` is the file line (from 1) of the problem, when the YAML reader gives one (§16.9).""" def __init__(self, message: str, line: int | None = None) -> None: @@ -1366,9 +1369,38 @@ def parse_item_content(text: str) -> list[Block]: return blocks -def parse_document(source: str, *, filename: str = "") -> Document: +def load(path: str | os.PathLike[str]) -> Document: + """Open the LegalDown file at *path* as a Document. + + The one call for a document that lives in a file: it reads the file as + UTF-8 and parses it as ``parse`` does, naming it in the result: + ``Document.filename`` is the file's name (without its directory) and + ``Document.path`` its absolute path, so a caller resolving files the document refers to + (``amends``, includes, attachments) knows where to look. Symbolic links + are not followed: the document is where it was named. + + Raises ``FileNotFoundError`` (or another ``OSError``) when the file cannot + be read, ``UnicodeDecodeError`` when it is not UTF-8, and + ``FrontmatterError`` when its frontmatter cannot be read. + """ + # The path as given is what is read: ``..`` after a symbolic link leads + # where the operating system takes it, which tidying the text of the path + # first could not tell. ``absolute`` makes it absolute and nothing more. + file = Path(path) + # Bytes, decoded once: a byte-order mark and the line endings (LF, CR and + # CRLF alike) are the parser's to read, so the text layer's own + # translation would only be a second, redundant pass over the file. + document = parse(file.read_bytes().decode("utf-8"), filename=file.name) + document.path = file.absolute() + return document + + +def parse(source: str, *, filename: str = "") -> Document: """Parse a LegalDown source string into a Document object. + For a file, ``load`` reads and parses it in one call. ``filename`` only + names the document in its diagnostics. + The parser is deliberately faithful to the source: nothing is rewritten to make a document valid, so the validator reports what the document actually says (a bare ``unit=M`` surfaces as duration-invalid-unit rather than being @@ -1413,6 +1445,18 @@ def parse_document(source: str, *, filename: str = "") -> Document: return document +def parse_document(source: str, *, filename: str = "") -> Document: + """Deprecated since 0.4.0, removed in 0.5.0. Same as ``parse``, with the same + arguments; ``load`` is the call for a file.""" + warnings.warn( + "legaldown.parse_document() is deprecated since 0.4.0 and will be removed in 0.5.0; " + "use legaldown.parse() for a string, or legaldown.load() for a file", + DeprecationWarning, + stacklevel=2, + ) + return parse(source, filename=filename) + + def collect_source_directives(document: Document) -> tuple[set[str], set[str]]: """Collect all ref and term targets used in a document. diff --git a/src/legaldown/positions.py b/src/legaldown/positions.py index a8c9621..6045973 100644 --- a/src/legaldown/positions.py +++ b/src/legaldown/positions.py @@ -1,7 +1,7 @@ """Where a parsed document's parts lie in its source, for diagnostics that name their line (§16.9). -``parse_document`` gives each document a ``SourceMap``; a document built from +``parse`` gives each document a ``SourceMap``; a document built from a dict has none, and its diagnostics carry no line. """ from __future__ import annotations diff --git a/src/legaldown/serializer.py b/src/legaldown/serializer.py index 15168d2..92a900e 100644 --- a/src/legaldown/serializer.py +++ b/src/legaldown/serializer.py @@ -186,7 +186,7 @@ def _paragraph_line(text: str, *, in_item: bool, alone: bool) -> str: Anything else that would open a block (a fence, a heading, an HTML block, a block quote, a thematic break, a list item) gets a backslash before it instead, which renders as nothing (CommonMark): only a model built in - code, not one from ``parse_document``, holds that, since the parser + code, not one from ``parse``, holds that, since the parser never turns the start of a block into paragraph text.""" if alone and (split := split_lone_tag(text)): return split diff --git a/src/legaldown/template.py b/src/legaldown/template.py new file mode 100644 index 0000000..f509d2e --- /dev/null +++ b/src/legaldown/template.py @@ -0,0 +1,463 @@ +"""Templates and their forms (LegalDown §15.7): a template read once, and what +is asked of the person filling it, given what they have answered so far. + +``load_template`` (or ``parse_template``) reads a template and the files it +includes, once. ``Template.form(answers)`` is a snapshot of the interview: +the questions reached so far, which of them are unanswered, which stop +assembly, what is wrong with the answers given, and whether assembly can run. +It is a plain value: change the answers, ask for a new form. +""" +from __future__ import annotations + +import math +import os +from collections.abc import Callable, Mapping +from dataclasses import dataclass, field +from datetime import date +from functools import cached_property +from pathlib import Path +from typing import Any + +import yaml + +from .assembly import ( + _MISSING, + AssemblyResult, + Question, + _answer_pairs, + _answer_problem, + _Answers, + _decide, + _effective_type, + _emit_result, + _FormAnswers, + _missing_diagnostic, + _questions, + _read, + _Template, +) +from .directives import PLACEHOLDER_TYPE_PARAMS +from .files import LoadFile, file_loader +from .validator import validate +from .validator.patterns import VALID_DURATION_UNITS +from .validator.result import Diagnostic, ValidationResult + +__all__ = ["AnswersError", "Form", "Template", "load_answers", "load_template", "parse_template"] + +#: The rules of the validator that a template must satisfy for assembly +#: (§15.7.2 gives assembly a template that validates without Errors): those of +#: questions, conditions, ``{{choose:}}``, placeholders and insertions. Any +#: other Error of ``Template.validation`` is advice — the validator reads a +#: template alone, without the files it includes, so a reference to a section +#: of a fragment is ``ref-broken`` to it while assembly reads the fragment. +TEMPLATE_RULES: frozenset[str] = frozenset({ + "question-invalid", + "condition-invalid", + "choose-invalid", + "placeholder-id-malformed", + "placeholder-in-structural-field", + "placeholder-question-mismatch", + "placeholder-type-inconsistent", + "placeholder-type-invalid", + "insertion-boundary", + "def-term-variable", + "drafting-note-def", + "template-fragment-invalid", +}) + +#: A rule of the validator that a template must satisfy, unless it includes +#: fragments: it then cannot tell a reference to a section of a fragment from +#: one to a section its condition removes. +_ALONE_ONLY: frozenset[str] = frozenset({"condition-reference-unsafe"}) + + +def _coerced(question: Question, value: Any) -> Any: + """*value* as an answer to *question*, if it can be read as one without guessing.""" + if isinstance(value, str): + if not value: + return value + try: + answer = question.from_text(value) + except ValueError: + return value + return value if answer is None else answer + if question.type == "money": + try: + if type(value) is int: + return str(value) + if isinstance(value, dict) and set(value) == {"amount", "currency"} and type(value["amount"]) is int: + return {**value, "amount": str(value["amount"])} + except ValueError: # an integer too long for Python to write as text + pass + return value + + +class AnswersError(ValueError): + """An answers file that is not an answers set (§15.7.1): not YAML, or not a + mapping of question ids to answers.""" + + +def load_answers(path: str | os.PathLike[str]) -> dict[str, Any]: + """The answers set (§15.7.1) in the YAML file at *path*: a mapping of question + ids to answers, empty for an empty file. Plain YAML: ``Template.coerce`` + puts right what it gets wrong for the template's questions. + + Raises ``OSError`` when the file cannot be read, and ``AnswersError`` when it + is not UTF-8, not YAML (a date such as ``2026-13-45`` is not one), or not a + mapping.""" + file = Path(path) + try: + answers = yaml.safe_load(file.read_bytes().decode("utf-8-sig")) + except (yaml.YAMLError, ValueError, RecursionError) as exc: # UnicodeDecodeError is a ValueError + raise AnswersError(f"cannot read the answers in {file}: {exc}") from exc + if answers is None: + return {} + if not isinstance(answers, dict): + raise AnswersError(f"the answers in {file} must be a YAML mapping of question ids to answers") + return answers + + +def _text(value: Any) -> str: + """``str(value)``, where an integer too long for Python to write is a note.""" + try: + return str(value) + except ValueError: + return "" + + +_JSON_NODES = 10_000 +_JSON_DEPTH = 40 + + +def _jsonable(value: Any, budget: list[int] | None = None, depth: int = 0) -> Any: + """*value* as strict JSON holds it: a date as its ISO text, a map with text + keys, a number that is not finite (or too long to write) as text. An answer + is data a person or a file gave: what is nested too deep, or runs past a + budget of nodes (an alias bomb in a YAML file), is cut short, so that + describing it stays cheap.""" + budget = [_JSON_NODES] if budget is None else budget + budget[0] -= 1 + if budget[0] < 0 or depth > _JSON_DEPTH: + return "..." + if isinstance(value, date): + return value.isoformat() + if isinstance(value, dict): + return {_text(key): _jsonable(item, budget, depth + 1) for key, item in value.items()} + if isinstance(value, (set, frozenset)): + return [_jsonable(item, budget, depth + 1) for item in sorted(value, key=_text)] + if isinstance(value, (list, tuple)): + return [_jsonable(item, budget, depth + 1) for item in value] + if isinstance(value, float): + return value if math.isfinite(value) else str(value) + if isinstance(value, int) and not isinstance(value, bool): + text = _text(value) + return text if text.startswith("<") else value + if value is None or isinstance(value, (str, bool)): + return value + return _text(value) + + +def _diagnostic_dict(diagnostic: Diagnostic) -> dict[str, Any]: + return {"rule": diagnostic.rule, "level": diagnostic.level, "message": diagnostic.message, "line": diagnostic.line} + + +def _placeholder_problems(t: _Template) -> list[Diagnostic]: + """A placeholder that would fill in a directive the validator rejects: a + repeated parameter, or a duration unit that §10.5 does not define. Those + in drafting notes are removed with them, and are no problem.""" + found = [(source.notes, occ) for source in (t.main, *t.subs.values()) for occ in source.occurrences] + found += [(set(), occ) for occ in (t.front.occurrences if t.front else [])] + problems = [] + for notes, occ in found: + directive = occ.directive + if directive.name != "placeholder" or occ.line in notes: + continue + for param in dict.fromkeys(directive.duplicates): + problems.append(Diagnostic( + "directive-duplicate-param", "error", + f"'{directive.source}' repeats the parameter '{param}'; the template is not assembled.", + )) + unit = directive.params.get(PLACEHOLDER_TYPE_PARAMS["duration"]) + if _effective_type(directive, t.declared) == "duration" and unit is not None and unit not in VALID_DURATION_UNITS: + problems.append(Diagnostic( + "duration-invalid-unit", "error", + f"'{directive.source}' fixes the duration unit '{unit}', which §10.5 does not define; " + f"the template is not assembled.", + )) + return problems + + +def _snapshot(resolve: LoadFile | None) -> LoadFile | None: + """*resolve* asking each file once, so that reading the template and + validating it see the same text.""" + if resolve is None: + return None + seen: dict[str, str | None] = {} + + def read(path: str) -> str | None: + if path not in seen: + seen[path] = resolve(path) + return seen[path] + + return read + + +class Template: + """A template: its source, the include fragments and LegalDown attachment + files it reads, and its questions. Read once; ``form`` asks and assembles + as often as needed. A template is a snapshot: files that change on disk + afterwards are not seen — load it again. + + *check* is whether ``problems`` include the validator's template rules + (``TEMPLATE_RULES``); ``load_template`` and ``parse_template`` always do. + + Raises ``FrontmatterError`` when the template's frontmatter cannot be read. + """ + + def __init__( + self, text: str, *, resolve: LoadFile | None = None, path: Path | None = None, check: bool = True + ) -> None: + self.path = path + self._resolve = _snapshot(resolve) + self._t = _read(text, self._resolve) + self._check = check + + @cached_property + def questions(self) -> tuple[Question, ...]: + """Every question (§15.2): the declared ones in declaration order, then + each undeclared placeholder, in the order first written.""" + return tuple(_questions(self._t)) + + @cached_property + def validation(self) -> ValidationResult: + """What ``validate`` says of the template read alone. Advice: only the + rules in ``problems`` stop assembly.""" + return validate(self._t.document, resolve=self._resolve) + + @cached_property + def problems(self) -> tuple[Diagnostic, ...]: + """Why the template cannot be assembled, whatever the answers: a file + it includes that cannot be read, a malformed placeholder, a translation + group, and the template rules (``TEMPLATE_RULES``) the validator reports + as Errors. Empty when it can.""" + t = self._t + problems = list(t.problems) + if self._check: + includes = any(include.path in t.subs for include in t.main.includes) + rules = TEMPLATE_RULES if includes else TEMPLATE_RULES | _ALONE_ONLY + problems += [d for d in self.validation.diagnostics if d.level == "error" and d.rule in rules] + problems += _placeholder_problems(t) + return tuple(problems) + + def form(self, answers: Mapping[str, Any] | None = None) -> Form: + """The form for *answers*, the answers so far (copied: the form does not + change when they do).""" + return _build_form(self, {} if answers is None else answers) + + def coerce(self, answers: Mapping[str, Any]) -> dict[str, Any]: + """*answers* with the shapes that plain YAML or JSON get wrong put right, + as a new mapping; *answers* is not changed, and nothing is raised. + + For a question of the template: text is read as ``Question.from_text`` + reads it (``yes`` for a boolean, ``5000 EUR`` for money, a choice by its + label, surrounding spaces dropped), except the empty text, which stays; + a money amount written as a number (``fee: 5000``, bare or in the + ``amount`` of a map) becomes the string the specification wants. + Anything it cannot put right — a float, a ``yes`` that YAML made ``True`` + for a text question, an unknown id — is left as it is, for ``form`` to + report. A form does not do this by itself: its diagnostics are the + specification's. A front end that takes answers from a person or a file + calls it first. Anything but a mapping is returned unchanged.""" + if not isinstance(answers, Mapping): + return answers # type: ignore[return-value] + by_id = {question.id: question for question in self.questions} + return { + key: _coerced(by_id[key], value) if isinstance(key, str) and key in by_id else value + for key, value in answers.items() + } + + +@dataclass(slots=True, frozen=True) +class Form: + """What is asked of the person filling a template, given the answers so + far (``Template.form``); a snapshot. + + ``questions`` are those reached now, in the order they are reached, + answered or not: a decision question when a condition or ``{{choose:}}`` + using it lies in a present unit, a value question when one of its + placeholders does. A unit under a condition whose question is unanswered + is not reached yet, so the form grows as decisions are made. + + ``unanswered`` are the reached questions with no valid answer and no + default; ``blocking`` the decision questions among them, which stop + assembly (§15.7.2 step 1), while an unanswered value question leaves its + blank. An answer that is not valid counts as no answer. ``unused`` are the + questions with an answer that are not reached: advisory, since one answer + that changes can make many unused. + + ``diagnostics`` say what is wrong with the answers; ``ready`` is whether + assembly can run (the template has no ``problems``, no answer is in + error) and ``complete`` whether nothing is left blank: ready, and no + question unanswered. + """ + + questions: tuple[Question, ...] + unanswered: tuple[Question, ...] + blocking: tuple[Question, ...] + unused: tuple[str, ...] + diagnostics: tuple[Diagnostic, ...] + ready: bool + complete: bool + _template: Template = field(repr=False, compare=False) + _answers: dict = field(repr=False, compare=False) + _resolved: _FormAnswers = field(repr=False, compare=False) + _decision: Any = field(repr=False, compare=False) + _pairs: tuple = field(repr=False, compare=False) + + def problem(self, qid: str) -> str | None: + """Why the answer to question *qid* — the answer given, else its + default — is not valid, or None.""" + answer = _Answers(self._answers, self._template._t.declared).get(qid) + if answer is _MISSING: + return None + return _answer_problem(self._template._t, qid, answer) + + def as_dict(self) -> dict[str, Any]: + """The form as plain data — JSON-ready — for a front end, a service or an + agent: ``ready``, ``complete``, the template's ``problems``, the + ``diagnostics`` about the answers (each with the ``question`` it is about, + ``""`` for none), and ``questions``: those reached first, in order, then + the others, each with + + - ``id``, ``type``, ``label`` (its prompt, else its id), ``prompt``, + ``declared``, ``choices``, ``currency``, ``unit``; + - ``reached`` and ``blocking`` (a decision that stops assembly); + - ``state``: ``answered`` (a valid answer), ``default`` (the default + applies), ``invalid`` (an answer that is not valid; it counts as none) + or ``unanswered``; ``answer`` (the answer given, else null), ``default``, + and ``answer_text``, ``default_text`` as a person would type them; + - ``problem``: why the answer is not valid, else null; + - ``hint``, in words, and ``accepts``, as data (``Question.accepts``). + """ + t = self._template._t + reached = {question.id for question in self.questions} + blocking = {question.id for question in self.blocking} + order = [*self.questions, *(q for q in self._template.questions if q.id not in reached)] + questions = [] + for question in order: + given = self._answers.get(question.id) + problem = self.problem(question.id) + if given is not None: + state = "invalid" if _answer_problem(t, question.id, given) else "answered" + else: + state = "default" if question.default is not None else "unanswered" + questions.append({ + "id": question.id, + "type": question.type, + "label": question.prompt or question.id, + "prompt": question.prompt, + "declared": question.declared, + "reached": question.id in reached, + "blocking": question.id in blocking, + "state": state, + "answer": _jsonable(given), + "answer_text": question.to_text(given), + "default": _jsonable(question.default), + "default_text": question.to_text(question.default), + "choices": dict(question.choices), + "currency": question.accepts.get("currency"), + "unit": question.accepts.get("unit"), + "problem": problem, + "hint": question.hint, + "accepts": question.accepts, + }) + return { + "ready": self.ready, + "complete": self.complete, + "problems": [_diagnostic_dict(d) for d in self._template.problems], + "diagnostics": [{**_diagnostic_dict(d), "question": qid} for qid, d in self._pairs], + "questions": questions, + } + + def assemble(self) -> AssemblyResult: + """Assemble the template with these answers (§15.7.2). Nothing is + assembled while the template has ``problems`` or an answer is in error; + the result then holds the diagnostics alone.""" + template = self._template + if template.problems: + return AssemblyResult(diagnostics=list(template.problems)) + return _emit_result(template._t, self._resolved, self._decision, list(self.diagnostics)) + + +def _copied(value: Any, seen: dict[int, tuple[Any, Any]] | None = None) -> Any: + """*value* with its containers copied, so that it does not change when the + caller's does; anything else is kept as it is (an answer of no valid kind is + reported, not copied). A container met again is the copy already made; the + memo keeps each original alive, so that its id cannot be taken by another.""" + seen = {} if seen is None else seen + if isinstance(value, (dict, list)): + if id(value) in seen: + return seen[id(value)][1] + if isinstance(value, dict): + made: Any = {} + seen[id(value)] = (value, made) + for key, item in value.items(): + made[key] = _copied(item, seen) + else: + made = [] + seen[id(value)] = (value, made) + made.extend(_copied(item, seen) for item in value) + return made + return value + + +def _build_form(template: Template, answers: Mapping[str, Any]) -> Form: + t = template._t + try: + given = _copied(dict(answers)) + except RecursionError: # an answer nested too deep to copy is reported as invalid, as it is kept + given = dict(answers) + resolved = _FormAnswers(given, t) + decision = _decide(t, resolved, inline=True) + by_id = {question.id: question for question in template.questions} + reached = tuple(by_id[qid] for qid in decision.needed if qid in by_id) + unanswered = tuple(question for question in reached if resolved.get(question.id) is _MISSING) + blocking = tuple(by_id[qid] for qid in decision.needed if qid in decision.missing and qid in by_id) + pairs = list(_answer_pairs(t, given)) + invalid = {qid for qid, d in pairs if d.rule == "answer-invalid"} + pairs += [(qid, _missing_diagnostic(qid)) for qid in decision.missing if qid not in invalid] + reached_ids = {question.id for question in reached} + diagnostics = tuple(d for _qid, d in pairs) + ready = not template.problems and not any(d.level == "error" for d in diagnostics) + return Form( + questions=reached, + unanswered=unanswered, + blocking=blocking, + unused=tuple(qid for qid in t.types if given.get(qid) is not None and qid not in reached_ids), + diagnostics=diagnostics, + ready=ready, + complete=ready and not unanswered, + _template=template, + _answers=given, + _resolved=resolved, + _decision=decision, + _pairs=tuple(pairs), + ) + + +def parse_template(text: str, *, resolve: Callable[[str], str | None] | None = None) -> Template: + """A ``Template`` from its source *text*. *resolve* reads the include + fragments and LegalDown attachment files it names — a relative path to the + file's text, or ``None`` when there is none (``file_loader`` for files on + disk). Without it, a template with either is refused (``Template.problems``): + reading them is a Full capability (§17.6).""" + return Template(text, resolve=resolve) + + +def load_template(path: str | os.PathLike[str]) -> Template: + """The ``Template`` in the file at *path*, with the include fragments and + attachment files it names, read from beside it (``file_loader``). The file is + read as written — byte-order mark and line endings included — since assembly + keeps every byte it does not edit (§15.7.2).""" + file = Path(path).absolute() + return Template(file.read_bytes().decode("utf-8"), resolve=file_loader(file.parent), path=file) diff --git a/src/legaldown/validator/__init__.py b/src/legaldown/validator/__init__.py index f2362f4..660908c 100644 --- a/src/legaldown/validator/__init__.py +++ b/src/legaldown/validator/__init__.py @@ -2,7 +2,7 @@ from __future__ import annotations from .conditions import ALWAYS, Condition, Presence, condition_problem, exclusive, parse_condition -from .core import AttachmentDefinitionsImporter, DefinitionsImporter, is_template, validate_document +from .core import AttachmentDefinitionsImporter, DefinitionsImporter, is_template, validate, validate_document from .helpers import ( format_section_number, is_positive_numeric, @@ -19,18 +19,21 @@ VALID_DURATION_UNITS, VALID_PLACEHOLDER_TYPES, ) -from .result import Diagnostic, PlacedMarker, SectionIndexEntry, ValidationResult +from .result import Diagnostic, DocumentIndex, InlineValues, PlacedMarker, SectionIndexEntry, ValidationResult from .templates import is_drafting_note __all__ = [ # Core - "validate_document", + "validate", + "validate_document", # deprecated since 0.4.0, removed in 0.5.0: use ``validate`` "is_template", "is_drafting_note", - "DefinitionsImporter", - "AttachmentDefinitionsImporter", + "DefinitionsImporter", # deprecated since 0.4.0, removed in 0.5.0 + "AttachmentDefinitionsImporter", # deprecated since 0.4.0, removed in 0.5.0 # Result types "ValidationResult", + "DocumentIndex", + "InlineValues", "SectionIndexEntry", "Diagnostic", "PlacedMarker", diff --git a/src/legaldown/validator/core.py b/src/legaldown/validator/core.py index b2a89f1..3e80d4a 100644 --- a/src/legaldown/validator/core.py +++ b/src/legaldown/validator/core.py @@ -1,13 +1,14 @@ """Core validation logic for LegalDown documents. Every diagnostic is recorded under the **stable rule id** the specification -assigns to its check (§16.1) via ``ValidationResult.error/warning/info``, +assigns to its check (§16.1) through the recorder's ``error/warning/info``, so tooling can filter, suppress, or escalate specific rules consistently across implementations (§16.9). """ from __future__ import annotations import re +import warnings from collections.abc import Callable, Iterator from dataclasses import replace from functools import cache, partial @@ -24,6 +25,7 @@ lex, mask_directives, ) +from ..files import LoadFile, file_loader, relative_path, within from ..markdown import HTML_COMMENT_RE, INLINE_HTML_RE, is_comment_only from ..models import LIST_KINDS, Amends, Document from ..positions import Locator @@ -47,7 +49,7 @@ VALID_DURATION_UNITS, VALID_PLACEHOLDER_TYPES, ) -from .result import Line, PlacedMarker, SectionIndexEntry, ValidationResult +from .result import Line, PlacedMarker, SectionIndexEntry, ValidationResult, _Recorder from .templates import ( BRACE_STRAY, DECISION_QUESTION_TYPES, @@ -198,7 +200,7 @@ def _frontmatter_fields( def _check_date_field( - label: str, value: str, questions: Any, rule: str, result: ValidationResult + label: str, value: str, questions: Any, rule: str, result: _Recorder ) -> None: """Report a frontmatter date field that is neither an ISO 8601 date nor a whole-value ``date`` placeholder (§3.10, §16.6).""" @@ -215,7 +217,7 @@ def _check_date_field( result.error(rule, f"{label} '{value}' must be a valid ISO 8601 date (YYYY-MM-DD).{hint}") -def _check_duration_unit(unit: str, result: ValidationResult) -> None: +def _check_duration_unit(unit: str, result: _Recorder) -> None: """Report a duration ``unit`` that §10.5 does not define.""" if not unit: result.error("duration-invalid-unit", "The duration unit is missing or empty.") @@ -239,7 +241,7 @@ def _check_duration_unit(unit: str, result: ValidationResult) -> None: def _check_directive_arguments( - directive: Directive, result: ValidationResult, *, effective_type: str | None = None + directive: Directive, result: _Recorder, *, effective_type: str | None = None ) -> bool: """Report the argument problems any directive can have; False if malformed. @@ -291,7 +293,7 @@ def _check_directive_arguments( def _check_placeholder( directive: Directive, - result: ValidationResult, + result: _Recorder, blanks: dict[str, Blank], questions: Any, *, @@ -312,7 +314,7 @@ def _check_placeholder( ptype = _placeholder_type(directive, declared) if not _check_directive_arguments(directive, result, effective_type=ptype): return - result.inline_placeholders.append((pid, ptype)) + result.index.values.placeholders.append((pid, ptype)) if not pid or not IDENTIFIER_RE.fullmatch(pid): result.error( "placeholder-id-malformed", @@ -367,7 +369,7 @@ def _check_placeholder( _check_duration_unit(code, result) -def _check_blank_codes(blanks: dict[str, Blank], result: ValidationResult) -> None: +def _check_blank_codes(blanks: dict[str, Blank], result: _Recorder) -> None: """Report each blank whose occurrences fix two currencies or units: one blank cannot hold two (§10.7).""" for pid, blank in blanks.items(): @@ -390,7 +392,7 @@ def _check_final( notes: list[tuple[Quote, Line]], conditions: list[tuple[str, str, Line]], questions: Any, - result: ValidationResult, + result: _Recorder, questions_line: int | None, ) -> None: """The final check (§15.9): no blank and no template construct remains @@ -471,12 +473,12 @@ def _inline_html(text: str, lexed: Lexed) -> str | None: def _check_raw_html( - document: Document, lex_fragment: Callable[[str], Lexed], result: ValidationResult, where: Locator + document: Document, lex_fragment: Callable[[str], Lexed], result: _Recorder, where: Locator ) -> None: """Warn about raw HTML other than comments (§8.7), which renderers leave out: once for each HTML block, in list items and quotes too, and once for each text — a paragraph, a table cell, a heading — holding inline HTML.""" - from ..definitions import block_fragments, nested_blocks # see the import note in validate_document + from ..definitions import block_fragments, nested_blocks # see the import note in _validate def warn(what: str, source: str, line: int | None) -> None: shown = source.strip().split("\n", 1)[0] @@ -509,7 +511,7 @@ def _check_never_true( markers: list[FoundMarker], units: Units, questions: Any, - result: ValidationResult, + result: _Recorder, where: Locator, ) -> None: """Report each conditional unit that can never appear: its own condition @@ -546,8 +548,8 @@ def _check_never_true( def is_template(document: Document) -> bool: """True if *document* is a template (§15.1): it declares ``questions``, carries a condition — on a section, an attachment, or a placed marker — - or contains a ``{{choose:}}``, wherever it is. ``validate_document`` - reports it too (``ValidationResult.is_template``).""" + or contains a ``{{choose:}}``, wherever it is. ``validate`` + reports it too (``DocumentIndex.is_template``).""" return _is_template(document, None, lex) @@ -558,7 +560,7 @@ def _is_template( ) -> bool: """``is_template``; *markers* are ``find_markers(document, lex_fragment)`` when the caller has them.""" - from ..definitions import text_fragments # see the import note in validate_document + from ..definitions import text_fragments # see the import note in _validate meta = document.metadata if markers is None: @@ -579,30 +581,124 @@ def _is_template( ) -def validate_document( +def validate( document: Document, *, + final: bool = False, + resolve: LoadFile | None = None, import_definitions: DefinitionsImporter | None = None, import_attachment_definitions: AttachmentDefinitionsImporter | None = None, - final: bool = False, ) -> ValidationResult: """Validate a LegalDown document and build lookup indices. Parameters ---------- document: - A ``Document`` instance (from ``legaldown.models``). - import_definitions: - Optional callback ``(amends_file, current_filename) -> dict | None`` - used to resolve definitions from an amended document. - import_attachment_definitions: - Optional callback ``(attachment_file) -> dict | None`` used to resolve - document-wide definitions declared inside an attachment file (§12.4). + A ``Document``, from ``load`` or ``parse``. final: Apply the final check (§15.9): the document is meant for signature, so a remaining blank or template construct is an Error. + resolve: + Reads a file the document refers to — the document it amends + (``amends.file``, §7.5) and its LegalDown attachment files (§12.4) — + for the definitions they declare: ``(relative_path) -> text | None``, + with ``None`` for a file that is not there or cannot be read. By + default the files are read from the directory of ``document.path`` + (``file_loader``), as far as that is within it (§2.3); a document + without a ``path`` (from ``parse``) has none, so the checks that need + them are skipped, as they are with ``resolve=lambda path: None``, for + a loaded document that is to be validated without reading anything. + Only the referenced file's own definitions are read, not those it + refers to in turn. + import_definitions, import_attachment_definitions: + Deprecated since 0.4.0, removed in 0.5.0: use *resolve*. They give the + definitions themselves, which *resolve* leaves to the validator. """ - result = ValidationResult() + resolve = _resolver(document, resolve, import_definitions, import_attachment_definitions, stacklevel=3) + return _validate(document, final, resolve, import_definitions, import_attachment_definitions) + + +def validate_document( + document: Document, + *, + import_definitions: DefinitionsImporter | None = None, + import_attachment_definitions: AttachmentDefinitionsImporter | None = None, + final: bool = False, +) -> ValidationResult: + """Deprecated since 0.4.0, removed in 0.5.0. Same as ``validate``, with the + same arguments.""" + warnings.warn( + "legaldown.validate_document() is deprecated since 0.4.0 and will be removed in 0.5.0; " + "use legaldown.validate()", + DeprecationWarning, + stacklevel=2, + ) + resolve = _resolver(document, None, import_definitions, import_attachment_definitions, stacklevel=3) + return _validate(document, final, resolve, import_definitions, import_attachment_definitions) + + +def _resolver( + document: Document, + resolve: LoadFile | None, + import_definitions: DefinitionsImporter | None, + import_attachment_definitions: AttachmentDefinitionsImporter | None, + *, + stacklevel: int, +) -> LoadFile | None: + """How the validation reads the files *document* refers to: *resolve*, else + the directory of its path, else not at all. + Says the importers are deprecated when one is given, and then reads nothing + by itself: each importer answers for its own files, as it always did.""" + if import_definitions is not None or import_attachment_definitions is not None: + warnings.warn( + "import_definitions and import_attachment_definitions are deprecated since 0.4.0 " + "and will be removed in 0.5.0; use validate(resolve=), which reads the files itself", + DeprecationWarning, + stacklevel=stacklevel, + ) + return resolve + if resolve is None and document.path is not None: + resolve = file_loader(document.path.parent) + + return resolve + + +def _definitions_in(document: Document, resolve: LoadFile | None, path: str) -> dict[str, str] | None: + """The definitions the file at *path* declares (``{id: term}``), or None + when it cannot be read as a LegalDown document: not there, no *resolve*, + or frontmatter that cannot be read (that file's problem, not this + document's). Any other exception is the parser's fault, as it is anywhere.""" + # Imported here for the reason given in ``_validate``. + from ..definitions import collect_definitions, definition_lookup + from ..parser import FrontmatterError, parse + + # Asked for as assembly asks (§15.7): normalized, and only a path within + # the document's directory (§2.3). + relative = relative_path(path) + if resolve is None or relative is None: + return None + path = relative + # A document that names itself has nothing to add to itself. + if document.path is not None and within(document.path.parent, path) == document.path.resolve(): + return None + text = resolve(path) + if text is None: + return None + try: + other = parse(text) + except FrontmatterError: + return None + return definition_lookup(collect_definitions(other)) + + +def _validate( + document: Document, + final: bool, + resolve: LoadFile | None, + import_definitions: DefinitionsImporter | None, + import_attachment_definitions: AttachmentDefinitionsImporter | None, +) -> ValidationResult: + result = _Recorder() # Where each part is, for the line of each diagnostic (§16.9). where = Locator(document) # §3.2: a newer declared version draws a Warning, never a failure, and @@ -733,7 +829,7 @@ def named(text: str, name: str) -> list[Directive]: seen_side_names.add(side_name) # §3.6 display derivation: label, else name with hyphens # replaced by spaces and each word capitalized. - result.side_lookup[side_name] = ( + result.index.side_lookup[side_name] = ( side.label or side_name.replace("-", " ").title() ) @@ -777,7 +873,7 @@ def named(text: str, name: str) -> list[Directive]: f"A representative of party '{party_name}' is missing the required name.", line=where.key(*party_path, "representatives", rep_index, "name"), ) - result.party_lookup[party_name] = ( + result.index.party_lookup[party_name] = ( party.label or party.legal_name or party_name ) @@ -845,7 +941,7 @@ def named(text: str, name: str) -> list[Directive]: result.error("attachment-id-duplicate", f"Duplicate attachment id '{att.id}'.", line=where.key(*att_path)) else: attachment_presence.setdefault(att.id, []).append(presence) - result.attachment_lookup.setdefault(att.id, att.title or att.id) + result.index.attachment_lookup.setdefault(att.id, att.title or att.id) if not att.title: result.error( "attachment-title-empty", @@ -926,7 +1022,7 @@ def free_identifier(base: str, presence: Presence) -> str: for section_index, section in enumerate(document.sections): # An out-of-range level is an Error, but the section still gets an - # index entry: `result.sections` is positionally paired with + # index entry: `result.index.sections` is positionally paired with # `document.sections` by callers (renderers, the editor), so skipping # one here would shift every later section's number and drop the last # one from rendered output. Numbering clamps into the valid range. @@ -1027,10 +1123,10 @@ def free_identifier(base: str, presence: Presence) -> str: level=level, number=number, ) - result.sections.append(entry) + result.index.sections.append(entry) # Alternatives share an identifier: a reference resolves to # whichever is present after assembly; the index keeps the first. - result.section_lookup.setdefault(identifier, entry) + result.index.section_lookup.setdefault(identifier, entry) last_level = level # ── Markers in the body (§5.7, §12.2, §15.3) ── @@ -1084,7 +1180,7 @@ def marker_line(found: FoundMarker) -> int | None: ) else: anchors.setdefault(anchor_id, []).append(presence) - result.section_lookup.setdefault(anchor_id, result.sections[found.section]) + result.index.section_lookup.setdefault(anchor_id, result.index.sections[found.section]) # ── Definitions (§7) ── # The mandatory, first-positioned Definitions section is gone (§7.2). A @@ -1139,7 +1235,7 @@ def definition_line(ref: Any) -> int | None: ) continue declared_terms.setdefault(def_id, []).append((ref.term, ref.auto_id, presence)) - result.definition_lookup.setdefault(def_id, ref.term or id_term(def_id)) + result.index.definition_lookup.setdefault(def_id, ref.term or id_term(def_id)) # ── Definition source-form checks (§7.2 validation table) ── for block_section, block_index, block in document.iter_indexed_blocks(): @@ -1182,48 +1278,52 @@ def definition_line(ref: Any) -> int | None: _amends_is_legaldown = True if import_definitions is not None: imported = import_definitions(amends_file, document.filename) - if imported is not None: - _amends_import_succeeded = True - _imported_definitions = imported - for def_id in result.definition_lookup: - if def_id in _imported_definitions: - result.warning( - "amend-def-override", - f"Amendment redefines '{def_id}' which exists in the original document.", - line=next( - (definition_line(ref) for ref in definition_refs if ref.id == def_id), None - ), - ) - for def_id, term_text in _imported_definitions.items(): - result.definition_lookup.setdefault(def_id, term_text) - # The original is always in force (§7.5), even where - # the amendment redefines the term under a condition. - declared_terms.setdefault(def_id, []).append((term_text, False, ALWAYS)) + else: + imported = _definitions_in(document, resolve, amends_file) + if imported is not None: + _amends_import_succeeded = True + _imported_definitions = imported + for def_id in result.index.definition_lookup: + if def_id in _imported_definitions: + result.warning( + "amend-def-override", + f"Amendment redefines '{def_id}' which exists in the original document.", + line=next( + (definition_line(ref) for ref in definition_refs if ref.id == def_id), None + ), + ) + for def_id, term_text in _imported_definitions.items(): + result.index.definition_lookup.setdefault(def_id, term_text) + # The original is always in force (§7.5), even where + # the amendment redefines the term under a condition. + declared_terms.setdefault(def_id, []).append((term_text, False, ALWAYS)) # ── Attachment definition import (§7, §12.4) ── # A {{def:}} inside an attachment file registers a document-wide term; ids # must remain unique across the combined document (§16.10). - if import_attachment_definitions is not None: - for att_index, att in enumerate(document.metadata.attachments): - if not att.file.endswith(LEGALDOWN_EXTENSIONS): - continue + for att_index, att in enumerate(document.metadata.attachments): + if not att.file.endswith(LEGALDOWN_EXTENSIONS): + continue + if import_attachment_definitions is not None: att_defs = import_attachment_definitions(att.file) - if not att_defs: - continue - # Its definitions are present when the attachment is (§15.3). - presence = own_presence(att.when, questions) - for def_id, term_text in att_defs.items(): - earlier = [other for _term, _auto, other in declared_terms.get(def_id, [])] - if _clashes(presence, earlier, questions): - result.error( - "def-duplicate-id", - f"Definition id '{def_id}' from attachment '{att.id}' collides with " - f"another definition that can appear with it (§16.10, §15.4).", - line=where.key("attachments", att_index), - ) - else: - declared_terms.setdefault(def_id, []).append((term_text, False, presence)) - result.definition_lookup.setdefault(def_id, term_text) + else: + att_defs = _definitions_in(document, resolve, att.file) + if not att_defs: + continue + # Its definitions are present when the attachment is (§15.3). + presence = own_presence(att.when, questions) + for def_id, term_text in att_defs.items(): + earlier = [other for _term, _auto, other in declared_terms.get(def_id, [])] + if _clashes(presence, earlier, questions): + result.error( + "def-duplicate-id", + f"Definition id '{def_id}' from attachment '{att.id}' collides with " + f"another definition that can appear with it (§16.10, §15.4).", + line=where.key("attachments", att_index), + ) + else: + declared_terms.setdefault(def_id, []).append((term_text, False, presence)) + result.index.definition_lookup.setdefault(def_id, term_text) # ── Inline directive validation ── blanks: dict[str, Blank] = {} @@ -1329,7 +1429,7 @@ def definition_line(ref: Any) -> int | None: elif name == "term": term_targets.append((value, line)) elif name == "date": - result.inline_dates.append(value) + result.index.values.dates.append(value) if not is_valid_iso_date(value): result.error( "date-invalid", @@ -1337,7 +1437,7 @@ def definition_line(ref: Any) -> int | None: ) elif name == "money": currency = params.get("currency", "") - result.inline_money.append((value, currency)) + result.index.values.money.append((value, currency)) if not is_valid_money_amount(value): result.error( "money-invalid-amount", @@ -1356,7 +1456,7 @@ def definition_line(ref: Any) -> int | None: ) elif name == "duration": dur_unit = params.get("unit", "") - result.inline_durations.append((value, dur_unit)) + result.index.values.durations.append((value, dur_unit)) if not is_positive_numeric(value): result.error( "duration-invalid-value", @@ -1371,7 +1471,7 @@ def definition_line(ref: Any) -> int | None: "party-name-malformed", f"Party directive has invalid role value '{value}'. Must match [a-z][a-z0-9-]*.", line=line, ) - elif value not in result.party_lookup: + elif value not in result.index.party_lookup: result.error( "party-unknown", f"Party directive references unknown party: '{value}'.", line=line, @@ -1407,7 +1507,7 @@ def definition_line(ref: Any) -> int | None: "field-type-undeclared", f"Field type '{ftype}' is not declared in field_types.", line=line, ) - result.inline_fields.append((value, ftype)) + result.index.values.fields.append((value, ftype)) elif name == "choose": with result.at(line): check_choose(directive, questions, result) @@ -1419,7 +1519,7 @@ def definition_line(ref: Any) -> int | None: f"Attachment reference '{{{{attach: {value}}}}}' references undeclared attachment id.", line=line, ) for target, line in ref_targets: - if target in result.section_lookup: + if target in result.index.section_lookup: continue if target in attachment_ids: # §5.6: attachments live in the anchor namespace but are @@ -1433,7 +1533,7 @@ def definition_line(ref: Any) -> int | None: result.error("ref-broken", f"Broken section reference: '{target}'.", line=line) for target, line in term_targets: result.used_terms.add(target) - if target not in result.definition_lookup: + if target not in result.index.definition_lookup: if document.metadata.amends: if _amends_is_legaldown and _amends_import_succeeded: result.error( @@ -1563,7 +1663,7 @@ def definition_line(ref: Any) -> int | None: for ref in definition_refs: if ref.id in _warned_defs: continue - if ref.id in result.definition_lookup and ref.id not in result.used_terms: + if ref.id in result.index.definition_lookup and ref.id not in result.used_terms: _warned_defs.add(ref.id) result.warning( "def-unreferenced", @@ -1573,12 +1673,12 @@ def definition_line(ref: Any) -> int | None: # What a renderer builds from: the template decision and the markers # that apply, as every check above read them. - result.is_template = template - result.placed_markers = _placed_markers(document, markers, template, marker_line) + result.index.is_template = template + result.index.placed_markers = _placed_markers(document, markers, template, marker_line) if document.filename: result.diagnostics = [replace(d, file=document.filename) for d in result.diagnostics] - return result + return result.result() def _placed_markers( @@ -1588,7 +1688,7 @@ def _placed_markers( line: Callable[[FoundMarker], int | None], ) -> list[PlacedMarker]: """The markers of *markers* that apply (``PlacedMarker``).""" - from ..definitions import list_fragments # see the import note in validate_document + from ..definitions import list_fragments # see the import note in _validate placed = [] lists: dict[tuple[int | None, int], list[tuple[str, bool, tuple[int, ...]]]] = {} # each list walked once diff --git a/src/legaldown/validator/result.py b/src/legaldown/validator/result.py index 214bb04..2370169 100644 --- a/src/legaldown/validator/result.py +++ b/src/legaldown/validator/result.py @@ -31,7 +31,7 @@ class Diagnostic: part of a diagnostic that is stable across implementations and spec revisions (§16.9) — plus the severity level and human-readable message, and where it is (§16.9): the file, and the line (from 1) of what it - reports, which a document parsed from source has (``parse_document``) + reports, which a document parsed from source has (``parse``) and one built from a dict has not (None). Compare diagnostics by their fields, not with one built without a line. """ @@ -45,7 +45,7 @@ class Diagnostic: @dataclass(slots=True, frozen=True, kw_only=True) class PlacedMarker: """A marker in body text that is in a marker position and applies in - its document (§5.7, §15.3), as ``validate_document`` places it — the + its document (§5.7, §15.3), as ``validate`` places it — the marker of a top-level paragraph or of a list item's first paragraph. A section's own marker is its ``Section.identifier`` and ``condition``. @@ -86,35 +86,97 @@ class PlacedMarker: @dataclass(slots=True) -class ValidationResult: - """Output of ``validate_document``: collected diagnostics and indices. +class InlineValues: + """The field-spec values the checks met, as written, in document order: + ``dates`` are the date values alone, the others ``(value, qualifier)``. + Placeholders of the frontmatter are among them, and a placeholder whose + arguments are malformed is not.""" + dates: list[str] = field(default_factory=list) + #: ``(amount, currency)`` + money: list[tuple[str, str]] = field(default_factory=list) + #: ``(amount, unit)`` + durations: list[tuple[str, str]] = field(default_factory=list) + #: ``(value, field type)`` + fields: list[tuple[str, str]] = field(default_factory=list) + #: ``(placeholder id, type)`` + placeholders: list[tuple[str, str]] = field(default_factory=list) - ``diagnostics`` is the authoritative record (rule id + severity + - message, §16.9). ``errors`` / ``warnings`` / ``infos`` remain as plain - message lists for existing callers and stay in sync with it. - """ - diagnostics: list[Diagnostic] = field(default_factory=list) - errors: list[str] = field(default_factory=list) - warnings: list[str] = field(default_factory=list) - infos: list[str] = field(default_factory=list) + +@dataclass(slots=True) +class DocumentIndex: + """What validating a document resolved, besides what it found: the numbered + sections, the display text the references resolve to, the values in the + body, and the template facts. A renderer or a UI reuses it instead of + deriving it again. Values and markers are as written: one that is invalid + is reported in the diagnostics too, so check ``ValidationResult.is_valid`` + before relying on them.""" + #: The numbered section index, in document order (positionally paired with + #: ``Document.sections``). sections: list[SectionIndexEntry] = field(default_factory=list) + #: The sections by the identifier or path a ``{{ref:}}`` names. section_lookup: dict[str, SectionIndexEntry] = field(default_factory=dict) + #: Display text, by definition id, party name, side name, attachment id. definition_lookup: dict[str, str] = field(default_factory=dict) party_lookup: dict[str, str] = field(default_factory=dict) side_lookup: dict[str, str] = field(default_factory=dict) attachment_lookup: dict[str, str] = field(default_factory=dict) - used_terms: set[str] = field(default_factory=set) - inline_dates: list[str] = field(default_factory=list) - inline_money: list[tuple[str, str]] = field(default_factory=list) - inline_durations: list[tuple[str, str]] = field(default_factory=list) - inline_fields: list[tuple[str, str]] = field(default_factory=list) - inline_placeholders: list[tuple[str, str]] = field(default_factory=list) + values: InlineValues = field(default_factory=InlineValues) #: Whether the document is a template (§15.1): it declares questions, #: carries a condition, or holds a ``{{choose:}}``. is_template: bool = False #: The markers in body text that apply (``PlacedMarker``), in document #: order. placed_markers: list[PlacedMarker] = field(default_factory=list) + + +@dataclass(slots=True, kw_only=True) +class ValidationResult: + """Output of ``validate``: the diagnostics found, and the index built. + + ``diagnostics`` is the authoritative record (rule id + severity + + message, §16.9); ``errors`` / ``warnings`` / ``infos`` are its messages by + severity. ``index`` is everything else validating resolved (``DocumentIndex``). + """ + diagnostics: list[Diagnostic] = field(default_factory=list) + index: DocumentIndex = field(default_factory=DocumentIndex) + + def _messages(self, level: str) -> list[str]: + return [d.message for d in self.diagnostics if d.level == level] + + @property + def errors(self) -> list[str]: + """The messages of the Error-level diagnostics.""" + return self._messages("error") + + @property + def warnings(self) -> list[str]: + """The messages of the Warning-level diagnostics.""" + return self._messages("warning") + + @property + def infos(self) -> list[str]: + """The messages of the Info-level diagnostics.""" + return self._messages("info") + + def rules(self, level: str | None = None) -> set[str]: + """Rule ids present in the result, optionally filtered by *level*.""" + return { + d.rule for d in self.diagnostics if level is None or d.level == level + } + + @property + def is_valid(self) -> bool: + """True if no errors were found.""" + return not any(d.level == "error" for d in self.diagnostics) + + +@dataclass(slots=True) +class _Recorder(ValidationResult): + """A ``ValidationResult`` as the validator builds it: it records the + diagnostics, and what only the checks need. ``result()`` is what the + caller gets.""" + #: The ``{{term:}}`` targets met, for the unreferenced-definition check. + used_terms: set[str] = field(default_factory=set) #: The lines diagnostics are recorded at by default (``at``), innermost last. _lines: list[Line] = field(default_factory=list, repr=False, compare=False) @@ -139,27 +201,17 @@ def error(self, rule: str, message: str, *, line: Line = None) -> None: """Record an Error-level diagnostic under stable rule id *rule*, at *line* (``at``).""" self._record(rule, "error", message, line) - self.errors.append(message) def warning(self, rule: str, message: str, *, line: Line = None) -> None: """Record a Warning-level diagnostic under stable rule id *rule*, at *line* (``at``).""" self._record(rule, "warning", message, line) - self.warnings.append(message) def info(self, rule: str, message: str, *, line: Line = None) -> None: """Record an Info-level diagnostic under stable rule id *rule*, at *line* (``at``).""" self._record(rule, "info", message, line) - self.infos.append(message) - def rules(self, level: str | None = None) -> set[str]: - """Rule ids present in the result, optionally filtered by *level*.""" - return { - d.rule for d in self.diagnostics if level is None or d.level == level - } - - @property - def is_valid(self) -> bool: - """True if no errors were found.""" - return len(self.errors) == 0 + def result(self) -> ValidationResult: + """What was recorded, as the plain ``ValidationResult``.""" + return ValidationResult(diagnostics=self.diagnostics, index=self.index) diff --git a/src/legaldown/validator/templates.py b/src/legaldown/validator/templates.py index d4f5cdc..9f6ffee 100644 --- a/src/legaldown/validator/templates.py +++ b/src/legaldown/validator/templates.py @@ -25,7 +25,7 @@ VALID_DURATION_UNITS, VALID_PLACEHOLDER_TYPES, ) -from .result import Line, ValidationResult +from .result import Line, _Recorder _CURRENCY_RE = re.compile(r"[A-Z]{3}") _LINE_BREAK_RE = re.compile(r"[\r\n\v\f\x1c\x1d\x1e\x85\u2028\u2029]") @@ -191,7 +191,7 @@ def _choices_problem(choices: Any) -> str | None: def check_questions( questions: Any, blanks: dict[str, Blank], - result: ValidationResult, + result: _Recorder, *, template: bool, not_line_editable: list[str], @@ -385,7 +385,7 @@ def insertion_boundary_problem( BRACE_STRAY = "'{{' does not begin a directive and is literal text; write '\\{{' if that is intended (§11.4)." -def check_choose(directive: Directive, questions: Any, result: ValidationResult) -> None: +def check_choose(directive: Directive, questions: Any, result: _Recorder) -> None: """Report a ``{{choose:}}`` that does not list exactly the answers of a declared decision question (choose-invalid, §15.5), and any ``{{`` in its phrases, which is literal text (brace-stray).""" @@ -435,7 +435,7 @@ def check_choose(directive: Directive, questions: Any, result: ValidationResult) def check_template_body( document: Document, lex_fragment: Callable[[str], Lexed], - result: ValidationResult, + result: _Recorder, *, template: bool, where: Locator | None = None, @@ -539,7 +539,7 @@ def _check_insertion( masked: str, insertion: Directive, directives: list[Directive], - result: ValidationResult, + result: _Recorder, line: Line = None, ) -> None: """Report *insertion*, at *line*, if it is not kept apart from template text.""" @@ -553,7 +553,7 @@ def _check_insertion( def _check_fragments( - document: Document, includes: list[tuple[str, Line]], result: ValidationResult, where: Locator | None + document: Document, includes: list[tuple[str, Line]], result: _Recorder, where: Locator | None ) -> None: """The Core parts of template-fragment-invalid (§15.3): each fragment is included once, and each LegalDown attachment file is declared by one diff --git a/tests/conformance/test_legaldown_fixtures.py b/tests/conformance/test_legaldown_fixtures.py index a9454fc..4f3283b 100644 --- a/tests/conformance/test_legaldown_fixtures.py +++ b/tests/conformance/test_legaldown_fixtures.py @@ -30,9 +30,9 @@ import pytest import yaml -from legaldown import CAPABILITIES, assemble -from legaldown.parser import parse_document -from legaldown.validator import validate_document +from legaldown import CAPABILITIES, Template, assemble, parse_template +from legaldown.parser import parse +from legaldown.validator import validate FIXTURES_DIR = os.environ.get("LEGALDOWN_FIXTURES_DIR", "") @@ -155,7 +155,16 @@ def load(relative: str) -> str | None: def _assemble_case(template: Path, answers: Path): loaded = yaml.safe_load(_read(answers)) or {} - return assemble(_read(template), loaded, load_file=_loader(template.parent)) + with pytest.warns(DeprecationWarning): # assemble is deprecated for Template (0.4.0); it is still pinned + return assemble(_read(template), loaded, load_file=_loader(template.parent)) + + +def _assemble_case_with_a_form(template: Path, answers: Path): + """The same by way of ``Template`` and ``Form``, without the validator's template rules: + the form and its result.""" + loaded = yaml.safe_load(_read(answers)) or {} + form = Template(_read(template), resolve=_loader(template.parent), check=False).form(loaded) + return form, form.assemble() def _validate_file(path: Path, config: dict): @@ -163,8 +172,8 @@ def _validate_file(path: Path, config: dict): reports the answer rules (§16.12).""" if "answers" in config: return _assemble_case(path, path.parent / config["answers"]) - document = parse_document(path.read_text(encoding="utf-8"), filename=path.name) - return validate_document(document, final=bool(config.get("final"))) + document = parse(path.read_text(encoding="utf-8"), filename=path.name) + return validate(document, final=bool(config.get("final"))) @pytest.mark.parametrize("case", list(_iter_valid_cases())) @@ -244,16 +253,16 @@ def test_a_fixture_keeps_its_diagnostics_when_written_back(path: Path): from legaldown.parser import FrontmatterError try: - document = parse_document(_read(path), filename=path.name) + document = parse(_read(path), filename=path.name) except FrontmatterError: pytest.skip("unreadable frontmatter: nothing to write back") - again = parse_document(serialize_document(document), filename=path.name) + again = parse(serialize_document(document), filename=path.name) # How the source writes its YAML is not the model's (block style when # written back). assert replace(again.metadata, not_line_editable=[]) == replace(document.metadata, not_line_editable=[]) def rules(document): - return Counter((d.rule, d.level) for d in validate_document(document).diagnostics) + return Counter((d.rule, d.level) for d in validate(document).diagnostics) assert rules(again) == rules(document) @@ -305,4 +314,12 @@ def test_assembly_case_assembles_byte_for_byte(case: Path): assert result.ok, result.diagnostics assert {"template.lgd": result.output, **result.files} == _expected_tree(case) # The assembly guarantee (§15.7.4): the output has no Errors either. - assert not validate_document(parse_document(result.output)).errors + assert not validate(parse(result.output)).errors + # And a ``Template`` with its ``Form`` assembles the same bytes, and says assembly can run. + form, through_a_form = _assemble_case_with_a_form(case / "template.lgd", case / "answers.yaml") + assert form.ready and (through_a_form.output, through_a_form.files) == (result.output, result.files) + # ... and so does the public door, with the validator's template rules on: the fixtures are valid templates. + loaded = yaml.safe_load(_read(case / "answers.yaml")) or {} + public = parse_template(_read(case / "template.lgd"), resolve=_loader(case)).form(loaded) + assert public.ready, (public.diagnostics, public._template.problems) + assert (public.assemble().output, public.assemble().files) == (result.output, result.files) diff --git a/tests/test_assembly.py b/tests/test_assembly.py index d4c80b6..fc228fb 100644 --- a/tests/test_assembly.py +++ b/tests/test_assembly.py @@ -22,14 +22,60 @@ import pytest import yaml +import legaldown from legaldown import ( AssemblyResult, + FrontmatterError, + Template, assemble, needed_questions, - parse_document, + parse, + parse_template, template_questions, - validate_document, + validate, ) +from legaldown.assembly import frontmatter_diagnostic + +# The functions this module tests are deprecated for ``Template`` (0.4.0); they are what it pins. +pytestmark = pytest.mark.filterwarnings( + r"ignore:legaldown\.(assemble|needed_questions|template_questions)\(\) is deprecated:DeprecationWarning" +) + + + +def _form_assemble(template, answers, *, load_file=None): + """``assemble`` by way of a ``Template`` and its ``Form``, without the template rules of the validator; + it must say what the function says: whether assembly can run, and what it writes.""" + legacy = legaldown.assemble(template, answers, load_file=load_file) + try: + tpl = Template(template, resolve=load_file, check=False) + except FrontmatterError as exc: + return AssemblyResult(diagnostics=[frontmatter_diagnostic(exc)]) + form = tpl.form(answers) + result = form.assemble() + assert form.ready == result.ok == legacy.ok + assert (result.output, result.files) == (legacy.output, legacy.files) + assert sorted(d.rule for d in result.diagnostics) == sorted(d.rule for d in legacy.diagnostics) + return result + + +def _form_needed(template, answers, *, load_file=None): + return list(parse_template(template, resolve=load_file).form(answers).questions) + + +def _form_questions(template, *, load_file=None): + return list(parse_template(template, resolve=load_file).questions) + + +@pytest.fixture(params=["functions", "template"], autouse=True) +def _via(request, monkeypatch): + """Every test of this module runs twice: through the functions, and through ``Template`` and ``Form``, + which must assemble the same bytes and ask the same questions.""" + if request.param == "template": + monkeypatch.setitem(globals(), "assemble", _form_assemble) + monkeypatch.setitem(globals(), "needed_questions", _form_needed) + monkeypatch.setitem(globals(), "template_questions", _form_questions) + FIXTURES = Path(os.environ.get("LEGALDOWN_FIXTURES_DIR", "")) / "assembly" CASES = sorted(path.name for path in FIXTURES.iterdir() if path.is_dir()) if FIXTURES.is_dir() else [] @@ -109,9 +155,9 @@ def test_fixture_template_and_its_output_have_no_errors(name): """The fixtures' premise (a template without Errors) and the assembly guarantee (§15.7.4): its output has none either.""" template = (FIXTURES / name / "template.lgd").read_text(encoding="utf-8") - assert validate_document(parse_document(template)).errors == [] + assert validate(parse(template)).errors == [] result = _assemble_case(name) - assert validate_document(parse_document(result.output)).errors == [] + assert validate(parse(result.output)).errors == [] # ── Helpers for the focused tests ──────────────────────────────── @@ -394,7 +440,7 @@ def test_an_unanswered_declared_blank_keeps_its_type_inline(self): "{{placeholder: day, type=date}}.\n" ) assert "questions" not in result.output - assert validate_document(parse_document(result.output)).errors == [] + assert validate(parse(result.output)).errors == [] def test_an_undeclared_or_already_typed_blank_is_left_as_written(self): template = _template("A {{placeholder: a}} and {{placeholder: b, type=date}}.\n") @@ -1041,7 +1087,7 @@ class TestHeadingBlocks: def test_a_blank_in_a_heading_block_is_filled(self, body): result = assemble(_template(body, _TEXT), {"name": "Acme"}) assert _body(result) == "\n" + body.replace("{{placeholder: name}}", "Acme") - assert "placeholder-unfilled" not in validate_document(parse_document(result.output)).rules() + assert "placeholder-unfilled" not in validate(parse(result.output)).rules() def test_a_marker_after_a_heading_block_is_no_condition(self): """An item's marker ends its first paragraph (§5.7): after a heading diff --git a/tests/test_cli.py b/tests/test_cli.py index 1b770d5..543b6e7 100644 --- a/tests/test_cli.py +++ b/tests/test_cli.py @@ -1,11 +1,14 @@ """CLI behavior: output formats, rule filtering, and exit codes.""" from __future__ import annotations +import io import json +import sys import pytest +import yaml -from legaldown import parse_document +from legaldown import load from legaldown.cli import EXIT_DIAGNOSTICS, EXIT_ERROR, EXIT_OK, main _VALID = """--- @@ -94,6 +97,13 @@ def test_missing_file_exits_two(tmp_path, capsys): assert "cannot read" in capsys.readouterr().err +def test_a_file_that_is_not_utf8_is_unreadable_not_a_crash(tmp_path, capsys): + path = tmp_path / "latin.lgd" + path.write_bytes(b"---\ntitle: caf\xe9\n---\n") + assert main(["validate", str(path)]) == EXIT_ERROR + assert "cannot read" in capsys.readouterr().err + + def test_malformed_frontmatter_is_reported_not_raised(write, capsys): path = write("bad.lgd", "---\ntitle: [unclosed\n---\n\n# Scope {#scope}\n") assert main(["validate", "--format", "json", str(path)]) == EXIT_DIAGNOSTICS @@ -122,12 +132,12 @@ def test_a_parser_fault_is_an_internal_error_not_a_diagnostic(write, capsys, mon # failure to report, and the other files are still validated. import legaldown.cli - def broken(source, *, filename=""): - if filename == "a.lgd": + def broken(path): + if path.name == "a.lgd": raise AttributeError("'set' object has no attribute 'get'") - return parse_document(source, filename=filename) + return load(path) - monkeypatch.setattr(legaldown.cli, "parse_document", broken) + monkeypatch.setattr(legaldown.cli, "load", broken) first, second = write("a.lgd", _VALID), write("b.lgd", _BROKEN) assert main(["validate", "--format", "json", str(first), str(second)]) == EXIT_ERROR captured = capsys.readouterr() @@ -260,3 +270,538 @@ def test_assemble_refuses_an_output_path_that_is_a_file(write, capsys, tmp_path) template.write_text(_VALID + "\n{{include: parts/a.lgd}}\n", encoding="utf-8") assert main(["assemble", str(template), "-o", str(tmp_path / "out")]) == EXIT_ERROR assert "the output is incomplete" in capsys.readouterr().err + + +def test_validate_reads_the_amended_original_beside_the_document(tmp_path, capsys): + amendment = ( + "---\ntitle: A\nsides:\n - name: s\n parties:\n - name: p\n" + " type: legal_entity\n legal_name: P\n" + "amends:\n title: Original\n file: original.lgd\n---\n\n# S {#s}\n\nUses {{term: services}}.\n" + ) + (tmp_path / "amendment.lgd").write_text(amendment, encoding="utf-8") + assert main(["validate", "--format", "json", str(tmp_path / "amendment.lgd")]) == EXIT_DIAGNOSTICS + rules = {d["rule"] for d in json.loads(capsys.readouterr().out)["diagnostics"]} + assert "amend-term-unresolvable" in rules # no original beside it: as before + (tmp_path / "original.lgd").write_text( + '---\ntitle: O\n---\n\n# S {#s}\n\n"Services" {{def: services}} means x.\n', encoding="utf-8" + ) + assert main(["validate", "--format", "json", str(tmp_path / "amendment.lgd")]) == EXIT_DIAGNOSTICS + diagnostics = json.loads(capsys.readouterr().out)["diagnostics"] + assert diagnostics # the document's own findings are still reported + assert not {"amend-term-unresolvable", "amend-term-undefined"} & {d["rule"] for d in diagnostics} + + +def test_a_fault_while_validating_is_an_internal_error_and_the_others_are_still_validated(write, capsys, monkeypatch): + import legaldown.cli + + real = legaldown.cli.validate + + def broken(document, **options): + if document.filename == "a.lgd": + raise AttributeError("boom") + return real(document, **options) + + monkeypatch.setattr(legaldown.cli, "validate", broken) + first, second = write("a.lgd", _VALID), write("b.lgd", _BROKEN) + assert main(["validate", "--format", "json", str(first), str(second)]) == EXIT_ERROR + captured = capsys.readouterr() + assert "ref-broken" in [d["rule"] for d in json.loads(captured.out)["diagnostics"]] # b.lgd + assert "internal error while validating" in captured.err and "AttributeError" in captured.err + + +def test_assemble_refuses_a_template_with_an_error_in_the_template_rules(write, capsys): + template = write("t.lgd", _TEMPLATE.replace("Hi {{placeholder: who}}.", 'Hi {{choose: x, true="only one phrase"}}.')) + answers = write("a.yaml", "x: true\n") + assert main(["assemble", str(template), "--answers", str(answers)]) == EXIT_DIAGNOSTICS + captured = capsys.readouterr() + assert captured.out == "" and "[choose-invalid]" in captured.err + + +def test_a_fault_while_assembling_is_an_internal_error(write, capsys, monkeypatch): + import legaldown.template + + def broken(self, answers=None): + raise AttributeError("boom") + + monkeypatch.setattr(legaldown.template.Template, "form", broken) + template = write("t.lgd", _TEMPLATE) + assert main(["assemble", str(template)]) == EXIT_ERROR + assert "internal error while assembling" in capsys.readouterr().err + + +def test_a_fault_while_reading_the_template_is_an_internal_error(write, capsys, monkeypatch): + import legaldown.cli + + def broken(path): + raise legaldown.AssemblyError("boom") + + monkeypatch.setattr(legaldown.cli, "load_template", broken) + assert main(["assemble", str(write("t.lgd", _TEMPLATE))]) == EXIT_ERROR + assert "internal error while reading" in capsys.readouterr().err + + +# ── legaldown questions ────────────────────────────────────────── + +_ASKING = _TEMPLATE.replace("questions:\n x:\n type: boolean\n", "questions:\n x:\n type: boolean\n prompt: Keep the extras?\n") + + +def _questions(args, capsys): + code = main(["questions", *args]) + return code, capsys.readouterr() + + +def test_questions_lists_what_is_asked_as_json(write, capsys): + template = write("t.lgd", _ASKING) + code, captured = _questions([str(template), "--format", "json"], capsys) + assert code == EXIT_OK + data = json.loads(captured.out) + assert (data["ready"], data["complete"], data["problems"]) == (False, False, []) + assert [q["id"] for q in data["questions"] if q["reached"]] == ["x", "who"] + by_id = {q["id"]: q for q in data["questions"]} + assert by_id["x"]["blocking"] and by_id["x"]["state"] == "unanswered" and by_id["x"]["label"] == "Keep the extras?" + assert by_id["who"]["hint"] == "text" and by_id["x"]["accepts"]["kind"] == "boolean" + assert [d["rule"] for d in data["diagnostics"]] == ["answer-missing"] + assert data["diagnostics"][0]["question"] == "x" + + +def test_questions_reads_the_answers_and_puts_their_shapes_right(write, capsys): + template = write("t.lgd", _ASKING) + answers = write("a.yaml", "x: no\nwho: Ann\n") + code, captured = _questions([str(template), "--answers", str(answers), "--format", "json"], capsys) + data = json.loads(captured.out) + assert code == EXIT_OK and data["ready"] and data["complete"] + assert {q["id"]: q["answer"] for q in data["questions"] if q["reached"]} == {"x": False, "who": "Ann"} + + +def test_questions_as_text(write, capsys): + template = write("t.lgd", _ASKING) + code, captured = _questions([str(template)], capsys) + assert code == EXIT_OK and captured.err == "" + lines = captured.out.splitlines() + assert lines[0].startswith("! x (boolean) Keep the extras?") and "enter yes or no" in lines[0] + assert lines[1].startswith("? who (text)") and "enter text" in lines[1] + assert "[answer-missing]" in captured.out and lines[-1] == "not ready" + answers = write("a.yaml", "x: true\nwho: Ann\n") + code, captured = _questions([str(template), "--answers", str(answers)], capsys) + assert captured.out.splitlines()[-1] == "complete" and "answered: Ann" in captured.out and "answered: yes" in captured.out + + +def test_questions_check_makes_not_ready_an_exit_status(write, capsys): + template = write("t.lgd", _ASKING) + assert _questions([str(template), "--check"], capsys)[0] == EXIT_DIAGNOSTICS + answers = write("a.yaml", "x: true\n") + assert _questions([str(template), "--answers", str(answers), "--check"], capsys)[0] == EXIT_OK # ready, with blanks + + +def test_questions_reports_a_template_with_problems(write, capsys): + template = write("t.lgd", _TEMPLATE.replace("Hi {{placeholder: who}}.", 'Hi {{choose: x, true="only one phrase"}}.')) + code, captured = _questions([str(template), "--format", "json"], capsys) + data = json.loads(captured.out) + assert code == EXIT_DIAGNOSTICS and [p["rule"] for p in data["problems"]] == ["choose-invalid"] + code, captured = _questions([str(template)], capsys) + assert code == EXIT_DIAGNOSTICS and "problem: [choose-invalid]" in captured.out + + +def test_questions_reports_unreadable_frontmatter_as_a_diagnostic(write, capsys): + template = write("t.lgd", "---\ntitle: [unclosed\n---\n") + code, captured = _questions([str(template), "--format", "json"], capsys) + data = json.loads(captured.out) + assert code == EXIT_DIAGNOSTICS and data["problems"][0]["rule"] == "frontmatter-invalid-yaml" and data["questions"] == [] + code, captured = _questions([str(template)], capsys) + assert code == EXIT_DIAGNOSTICS and "[frontmatter-invalid-yaml]" in captured.err + + +@pytest.mark.parametrize("answers", ["x: [unclosed\n", "- a\n- b\n"]) +def test_questions_with_an_unreadable_answers_file_or_template_is_an_error(write, capsys, tmp_path, answers): + template = write("t.lgd", _ASKING) + assert _questions([str(template), "--answers", str(write("a.yaml", answers))], capsys)[0] == EXIT_ERROR + assert _questions([str(tmp_path / "nope.lgd")], capsys)[0] == EXIT_ERROR + assert _questions([str(template), "--answers", str(tmp_path / "nope.yaml")], capsys)[0] == EXIT_ERROR + + +def test_assemble_puts_the_shapes_of_the_answers_file_right(write, capsys): + template = write("t.lgd", _ASKING.replace("Hi {{placeholder: who}}.", "Fee {{placeholder: fee, type=money}}.")) + answers = write("a.yaml", "x: no\nfee: 5000 eur\n") # not the shape of a money answer + assert main(["assemble", str(template), "--answers", str(answers)]) == EXIT_OK + assert "{{money: 5000, currency=EUR}}" in capsys.readouterr().out + integers = write("b.yaml", "x: no\nfee:\n amount: 5000\n currency: EUR\n") # a number as the amount + assert main(["assemble", str(template), "--answers", str(integers)]) == EXIT_OK + assert "{{money: 5000, currency=EUR}}" in capsys.readouterr().out + + +def test_questions_writes_utf8_whatever_the_terminal_encoding(tmp_path): + import os + import subprocess + import sys + + template = tmp_path / "t.lgd" + template.write_text(_TEMPLATE.replace("prompt: x", "x").replace(" type: boolean\n", " type: boolean\n prompt: Jméno — ano?\n", 1), encoding="utf-8") + src = os.path.join(os.path.dirname(__file__), "..", "src") + env = {**os.environ, "PYTHONIOENCODING": "ascii", "PYTHONPATH": src} + for fmt in ("text", "json"): + done = subprocess.run( + [sys.executable, "-m", "legaldown.cli", "questions", str(template), "--format", fmt], + capture_output=True, env=env, check=False, + ) + assert done.returncode == 0, done.stderr + assert "Jméno — ano?".encode() in done.stdout + + +def test_questions_json_is_strict_json_for_not_a_number_answers(write, capsys): + template = write("t.lgd", _ASKING) + answers = write("a.yaml", "x: true\nwho: .nan\n") + code, captured = _questions([str(template), "--answers", str(answers), "--format", "json"], capsys) + assert code == EXIT_OK + json.loads(captured.out, parse_constant=lambda name: pytest.fail(f"{name} is not JSON")) + + +def test_questions_with_answers_nested_too_deep_is_an_error(write, capsys): + template = write("t.lgd", _ASKING) + answers = write("a.yaml", "x: " + "[" * 3000 + "]" * 3000 + "\n") + code, captured = _questions([str(template), "--answers", str(answers)], capsys) + assert code == EXIT_ERROR and "cannot read the answers" in captured.err + + +def test_a_fault_while_listing_the_questions_is_an_internal_error(write, capsys, monkeypatch): + import legaldown.template + + def broken(self): + raise AttributeError("boom") + + monkeypatch.setattr(legaldown.template.Form, "as_dict", broken) + template = write("t.lgd", _ASKING) + assert _questions([str(template), "--format", "json"], capsys)[0] == EXIT_ERROR + + +# ── legaldown assemble -i, --save-answers ───────────────────────── + +_INTERVIEW = _TEMPLATE.replace( + " type: boolean\n", " type: boolean\n prompt: Keep the extras?\n", 1 +).replace("Hi {{placeholder: who}}.", "Hi {{placeholder: who}}. Fee {{placeholder: fee, type=money}}.") + + +class _Terminal(io.StringIO): + """Standard input that is a terminal, typing *lines*.""" + + def isatty(self): + return True + + +def _type(monkeypatch, *lines: str) -> None: + monkeypatch.setattr(sys, "stdin", _Terminal("".join(f"{line}\n" for line in lines))) + + +def test_an_interview_asks_for_the_answers_and_writes_the_document_to_stdout(write, capsys, monkeypatch): + template = write("t.lgd", _INTERVIEW) + _type(monkeypatch, "no", "Ann", "250 eur") + assert main(["assemble", str(template), "-i"]) == EXIT_OK + captured = capsys.readouterr() + assert "Hi Ann. Fee {{money: 250, currency=EUR}}." in captured.out and "Extra." not in captured.out + assert "Keep the extras? (yes or no)" in captured.err and "who (text)" in captured.err # prompts: stderr + assert "Keep the extras?" not in captured.out + + +def test_a_decision_is_asked_again_until_it_is_answered(write, capsys, monkeypatch): + template = write("t.lgd", _INTERVIEW) + _type(monkeypatch, "", "maybe", "yes", "Ann", "", ) + assert main(["assemble", str(template), "-i"]) == EXIT_OK + err = capsys.readouterr().err + assert "An answer is needed." in err and "Enter yes or no." in err + assert err.count("Keep the extras?") == 3 + + +def test_a_value_question_may_be_left_to_its_blank(write, capsys, monkeypatch): + template = write("t.lgd", _INTERVIEW) + _type(monkeypatch, "yes", "", "", "") + assert main(["assemble", str(template), "-i"]) == EXIT_OK + out = capsys.readouterr().out + assert "{{placeholder: who}}" in out and "Extra." in out + + +def test_an_interview_shows_the_default_and_the_questions_the_answers_file_gave_are_not_asked(write, capsys, monkeypatch): + template = write( + "t.lgd", + _INTERVIEW.replace(" prompt: Keep the extras?\n", " prompt: Keep the extras?\n fee:\n type: money\n default:\n amount: '100'\n currency: EUR\n", 1), + ) + answers = write("a.yaml", "x: no\n") + _type(monkeypatch, "Ann", "") + assert main(["assemble", str(template), "--answers", str(answers), "-i"]) == EXIT_OK + err = capsys.readouterr().err + assert "[100 EUR]" in err and "Keep the extras?" not in err + + +def test_an_answer_the_file_gave_that_is_not_valid_is_asked_again(write, capsys, monkeypatch): + template = write("t.lgd", _INTERVIEW) + answers = write("a.yaml", "x: maybe\nwho: Ann\n") + _type(monkeypatch, "no", "") + assert main(["assemble", str(template), "--answers", str(answers), "-i"]) == EXIT_OK + assert "is not valid" in capsys.readouterr().err + + +def test_without_a_terminal_it_lists_what_must_be_answered(write, capsys): + template = write("t.lgd", _INTERVIEW) + assert main(["assemble", str(template), "-i"]) == EXIT_DIAGNOSTICS # pytest's stdin is no terminal + captured = capsys.readouterr() + assert captured.out == "" and "x: Keep the extras? (enter yes or no)" in captured.err and "-i" in captured.err + answers = write("a.yaml", "x: no\n") + assert main(["assemble", str(template), "--answers", str(answers), "-i"]) == EXIT_OK # nothing blocks: batch + assert "{{placeholder: who}}" in capsys.readouterr().out + + +def test_without_i_a_missing_decision_says_how_to_be_asked(write, capsys): + template = write("t.lgd", _INTERVIEW) + assert main(["assemble", str(template)]) == EXIT_DIAGNOSTICS + err = capsys.readouterr().err + assert "[answer-missing]" in err and "-i" in err + + +def test_the_end_of_the_input_ends_the_interview(write, capsys, monkeypatch): + template = write("t.lgd", _INTERVIEW) + _type(monkeypatch) # nothing typed + assert main(["assemble", str(template), "-i"]) == EXIT_DIAGNOSTICS + captured = capsys.readouterr() + assert captured.out == "" and "[answer-missing]" in captured.err + + +def test_an_interruption_saves_the_answers_and_assembles_nothing(write, capsys, monkeypatch, tmp_path): + template = write("t.lgd", _INTERVIEW) + + class Typist(_Terminal): + def __init__(self): + super().__init__("no\n") + + def readline(self, *args): + line = super().readline(*args) + if not line: + raise KeyboardInterrupt + return line + + monkeypatch.setattr(sys, "stdin", Typist()) + saved = tmp_path / "saved.yaml" + assert main(["assemble", str(template), "-i", "--save-answers", str(saved)]) == EXIT_ERROR + captured = capsys.readouterr() + assert captured.out == "" and "interrupted" in captured.err + assert yaml.safe_load(saved.read_text(encoding="utf-8")) == {"x": False} + + +def test_saved_answers_give_the_same_document_when_read_back(write, capsys, monkeypatch, tmp_path): + template = write("t.lgd", _INTERVIEW) + saved = tmp_path / "out" / "saved.yaml" + _type(monkeypatch, "no", "Ann Čech", "250 eur") + assert main(["assemble", str(template), "-i", "--save-answers", str(saved)]) == EXIT_OK + first = capsys.readouterr().out + text = saved.read_text(encoding="utf-8") + assert "Ann Čech" in text and "'250'" in text # not escaped; an amount stays a quoted string + assert main(["assemble", str(template), "--answers", str(saved)]) == EXIT_OK + assert capsys.readouterr().out == first + + +def test_answers_are_saved_even_when_nothing_could_be_assembled(write, capsys, tmp_path): + template = write("t.lgd", _INTERVIEW) + answers = write("a.yaml", "who: Ann\n") + saved = tmp_path / "saved.yaml" + assert main(["assemble", str(template), "--answers", str(answers), "--save-answers", str(saved)]) == EXIT_DIAGNOSTICS + assert yaml.safe_load(saved.read_text(encoding="utf-8")) == {"who": "Ann"} + broken = write("b.lgd", "---\ntitle: [unclosed\n---\n") + other = tmp_path / "other.yaml" + assert main(["assemble", str(broken), "--answers", str(answers), "--save-answers", str(other)]) == EXIT_DIAGNOSTICS + assert yaml.safe_load(other.read_text(encoding="utf-8")) == {"who": "Ann"} + + +def test_answers_are_not_saved_when_the_answers_file_cannot_be_read(write, capsys, tmp_path): + template = write("t.lgd", _INTERVIEW) + saved = tmp_path / "saved.yaml" + assert main(["assemble", str(template), "--answers", str(tmp_path / "nope.yaml"), "--save-answers", str(saved)]) == EXIT_ERROR + assert not saved.exists() + + +def test_a_failure_to_save_the_answers_is_an_error(write, capsys, tmp_path): + template = write("t.lgd", _INTERVIEW) + answers = write("a.yaml", "x: no\nwho: Ann\n") + blocked = tmp_path / "blocked" + blocked.write_text("a file, not a directory", encoding="utf-8") + code = main(["assemble", str(template), "--answers", str(answers), "--save-answers", str(blocked / "a.yaml")]) + assert code == EXIT_ERROR and "cannot write the answers" in capsys.readouterr().err + + +def test_a_template_that_cannot_be_assembled_asks_nothing(write, capsys, monkeypatch): + template = write("t.lgd", _INTERVIEW.replace("Hi {{placeholder: who}}.", 'Hi {{choose: x, true="only one phrase"}}.')) + _type(monkeypatch, "no", "Ann") + assert main(["assemble", str(template), "-i"]) == EXIT_DIAGNOSTICS + err = capsys.readouterr().err + assert "[choose-invalid]" in err and "Keep the extras?" not in err + monkeypatch.setattr(sys, "stdin", io.StringIO("")) # no terminal + assert main(["assemble", str(template), "-i"]) == EXIT_DIAGNOSTICS + err = capsys.readouterr().err + assert "[choose-invalid]" in err and "must be answered" not in err + + +def test_the_empty_line_drops_an_answer_that_is_not_valid(write, capsys, monkeypatch, tmp_path): + template = write("t.lgd", _INTERVIEW) + answers = write("a.yaml", "x: no\nwho: ' '\nfee: 5000\n") + saved = tmp_path / "saved.yaml" + _type(monkeypatch, "", "") + code = main(["assemble", str(template), "--answers", str(answers), "-i", "--save-answers", str(saved)]) + assert code == EXIT_OK # the blanks stay; nothing invalid is left + assert "who" not in yaml.safe_load(saved.read_text(encoding="utf-8")) + + +def test_text_that_is_not_utf8_is_not_an_answer(write, capsys, monkeypatch): + template = write("t.lgd", _INTERVIEW) + _type(monkeypatch, "no", "\udcff\udcfe bad", "Ann", "") + assert main(["assemble", str(template), "-i"]) == EXIT_OK + captured = capsys.readouterr() + assert "Hi Ann." in captured.out and "UTF-8" in captured.err + + +def test_the_answers_are_saved_beside_and_moved_into_place(write, tmp_path): + from legaldown.cli import _save_answers + + saved = tmp_path / "a.yaml" + saved.write_text("# my notes\nwho: Old\n", encoding="utf-8") + assert _save_answers(saved, {"who": "New"}) is None + assert yaml.safe_load(saved.read_text(encoding="utf-8")) == {"who": "New"} + assert [p.name for p in tmp_path.iterdir()] == ["a.yaml"] # no temporary file left + assert "cannot write" in _save_answers(saved, {"who": object()}) + assert yaml.safe_load(saved.read_text(encoding="utf-8")) == {"who": "New"} # as it was + assert [p.name for p in tmp_path.iterdir()] == ["a.yaml"] + assert _save_answers(saved, {"who": "\udcff"}) is None # YAML writes it escaped, and it reads back + assert yaml.safe_load(saved.read_text(encoding="utf-8")) == {"who": "\udcff"} + + +def test_the_default_that_is_accepted_is_echoed(write, capsys, monkeypatch): + template = write( + "t.lgd", + _INTERVIEW.replace(" prompt: Keep the extras?\n", " prompt: Keep the extras?\n fee:\n type: money\n default:\n amount: '100'\n currency: EUR\n", 1), + ) + _type(monkeypatch, "no", "Ann", "") + assert main(["assemble", str(template), "-i"]) == EXIT_OK + assert "the default, 100 EUR" in capsys.readouterr().err + + +def test_keys_typed_as_escape_sequences_are_not_an_answer(write, capsys, monkeypatch): + template = write("t.lgd", _INTERVIEW) + _type(monkeypatch, "no", "abc\x1b[D\x1b[DX", "Ann", "") # the arrow keys of a terminal without line editing + assert main(["assemble", str(template), "-i"]) == EXIT_OK + captured = capsys.readouterr() + assert "Hi Ann." in captured.out and "\x1b" not in captured.out and "control character" in captured.err + + +# ── --save-answers keeps what it replaces ───────────────────────── + + +def test_saving_the_answers_keeps_the_permissions_of_the_file_it_replaces(write, tmp_path): + import os + import stat + + from legaldown.cli import _save_answers + + saved = tmp_path / "a.yaml" + saved.write_text("who: Old\n", encoding="utf-8") + os.chmod(saved, 0o640) + assert _save_answers(saved, {"who": "New"}) is None + assert stat.S_IMODE(saved.stat().st_mode) == 0o640 + fresh = tmp_path / "new" / "b.yaml" + assert _save_answers(fresh, {"who": "New"}) is None + mask = os.umask(0) + os.umask(mask) + assert stat.S_IMODE(fresh.stat().st_mode) == 0o666 & ~mask # not the 0600 of a temporary file + + +def test_saving_the_answers_writes_through_a_symbolic_link(tmp_path): + from legaldown.cli import _save_answers + + real = tmp_path / "real.yaml" + real.write_text("who: Old\n", encoding="utf-8") + link = tmp_path / "link.yaml" + try: + link.symlink_to(real) + except OSError: + pytest.skip("symbolic links are not available") + assert _save_answers(link, {"who": "New"}) is None + assert link.is_symlink() and yaml.safe_load(real.read_text(encoding="utf-8")) == {"who": "New"} + + +def test_the_answers_are_never_saved_over_the_template(write, capsys): + template = write("t.lgd", _INTERVIEW) + before = template.read_text(encoding="utf-8") + assert main(["assemble", str(template), "--save-answers", str(template)]) == EXIT_ERROR + assert "names the template itself" in capsys.readouterr().err + assert template.read_text(encoding="utf-8") == before + + +def test_an_answer_that_cannot_be_written_as_utf8_is_an_error_not_a_traceback(write, capsys): + template = write("t.lgd", _INTERVIEW) + answers = write("a.yaml", 'x: no\nwho: "\\uDCFF"\n') + assert main(["assemble", str(template), "--answers", str(answers)]) == EXIT_ERROR + captured = capsys.readouterr() + assert captured.out == "" and "cannot be written as UTF-8" in captured.err + code = main(["questions", str(template), "--answers", str(answers), "--format", "json"]) + assert code == EXIT_OK and "\\udcff" in capsys.readouterr().out # described, escaped + + +def test_the_end_of_the_input_is_said(write, capsys, monkeypatch): + template = write("t.lgd", _INTERVIEW) + _type(monkeypatch, "no") # who is not answered: the input ends + assert main(["assemble", str(template), "-i"]) == EXIT_OK + assert "End of the input" in capsys.readouterr().err + + +def test_nothing_read_and_nothing_typed_does_not_replace_the_file_to_save_to(write, capsys, monkeypatch, tmp_path): + keep = write("keep.yaml", "# mine\nwho: Ann\nx: false\n") + before = keep.read_text(encoding="utf-8") + # a template that cannot be read (a typo in its name) + assert main(["assemble", str(tmp_path / "typo.lgd"), "--save-answers", str(keep)]) == EXIT_ERROR + # an interview with no terminal and nothing typed + template = write("t.lgd", _INTERVIEW) + assert main(["assemble", str(template), "-i", "--save-answers", str(keep)]) == EXIT_DIAGNOSTICS + assert "was not written" in capsys.readouterr().err + # an interview interrupted before the first answer + class Interrupted(_Terminal): + def readline(self, *args): + raise KeyboardInterrupt + + monkeypatch.setattr(sys, "stdin", Interrupted()) + assert main(["assemble", str(template), "-i", "--save-answers", str(keep)]) == EXIT_ERROR + assert keep.read_text(encoding="utf-8") == before + + +def test_an_answers_file_that_was_read_is_saved_even_when_it_is_empty_of_answers(write, tmp_path): + answers = write("a.yaml", "") + template = write("t.lgd", _INTERVIEW) + saved = tmp_path / "saved.yaml" + assert main(["assemble", str(template), "--answers", str(answers), "--save-answers", str(saved)]) == EXIT_DIAGNOSTICS + assert yaml.safe_load(saved.read_text(encoding="utf-8")) == {} + + +def test_the_template_is_not_overwritten_through_a_hard_link_either(write, tmp_path): + import os + + template = write("t.lgd", _INTERVIEW) + link = tmp_path / "hard.lgd" + try: + os.link(template, link) + except OSError: + pytest.skip("hard links are not available") + before = template.read_text(encoding="utf-8") + assert main(["assemble", str(template), "--save-answers", str(link)]) == EXIT_ERROR + assert template.read_text(encoding="utf-8") == before + + +def test_a_symbolic_link_loop_as_the_file_to_save_to_is_an_error_not_a_traceback(write, capsys, tmp_path): + import os + + template = write("t.lgd", _INTERVIEW) + answers = write("a.yaml", "x: no\nwho: Ann\n") + loop = tmp_path / "loop.yaml" + try: + os.symlink(loop, loop) + except OSError: + pytest.skip("symbolic links are not available") + code = main(["assemble", str(template), "--answers", str(answers), "--save-answers", str(loop)]) + err = capsys.readouterr().err + if code == EXIT_ERROR: # Python before 3.13 cannot resolve a link loop: said, not a traceback + assert "cannot write the answers" in err + else: # Python 3.13 resolves it to itself, and the answers replace the loop + assert code == EXIT_OK and not loop.is_symlink() + assert yaml.safe_load(loop.read_text(encoding="utf-8")) == {"x": False, "who": "Ann"} diff --git a/tests/test_conditions.py b/tests/test_conditions.py index f08daf8..65233dd 100644 --- a/tests/test_conditions.py +++ b/tests/test_conditions.py @@ -6,8 +6,8 @@ from legaldown import serialize_document from legaldown.markers import Marker, parse_marker, split_heading -from legaldown.parser import parse_document -from legaldown.validator import validate_document +from legaldown.parser import parse +from legaldown.validator import validate _SIDES = """sides: - name: providers @@ -31,11 +31,18 @@ def _parse(body: str, frontmatter: str = _QUESTIONS): - return parse_document(f"---\ntitle: Fixture\n{_SIDES}{frontmatter}---\n\n{body}\n") + return parse(f"---\ntitle: Fixture\n{_SIDES}{frontmatter}---\n\n{body}\n") + + +def _original(**terms: str) -> str: + """A LegalDown file declaring the definitions *terms* (id=term), as an + amended original or an attachment file does.""" + declared = "".join(f'"{term}" {{{{def: {id}}}}} means x.\n\n' for id, term in terms.items()) + return f"---\ntitle: Original\n---\n\n# Terms\n\n{declared}" def _rules(body: str, frontmatter: str = _QUESTIONS, **options) -> set[str]: - return validate_document(_parse(body, frontmatter), **options).rules() + return validate(_parse(body, frontmatter), **options).rules() # ── Markers (§15.3) ─────────────────────────────────────────────── @@ -71,8 +78,8 @@ def test_a_conditional_heading_round_trips_and_its_identifier_is_generated(): section = document.sections[0] assert (section.title, section.identifier, section.condition) == ("Services", "", "vat") assert "\n# Services {when=vat}\n" in serialize_document(document) - result = validate_document(document) - assert result.sections[0].identifier == "services" + result = validate(document) + assert result.index.sections[0].identifier == "services" assert document.sections[0].identifier == "" # validation leaves the document as it was @@ -125,7 +132,7 @@ def test_a_unit_that_can_never_appear_is_a_warning(body): def test_only_the_contradicting_unit_is_reported(): body = "# A {when=vat}\n\n## B {when=!vat}\n\n### C {when=forum:courts}\n\nText." - result = validate_document(_parse(body)) + result = validate(_parse(body)) assert [d.rule for d in result.diagnostics].count("condition-never-true") == 1 @@ -190,21 +197,21 @@ def test_alternative_attachments_may_share_an_id(): def test_headings_that_never_appear_together_do_not_collide(): body = "# Services {when=vat}\n\nA.\n\n# Services {when=!vat}\n\nB." - result = validate_document(_parse(body)) - assert [entry.identifier for entry in result.sections] == ["services", "services"] + result = validate(_parse(body)) + assert [entry.identifier for entry in result.index.sections] == ["services", "services"] assert "anchor-autogen-collision" not in result.rules() def test_an_explicit_identifier_always_wins_even_later(): body = "# Scope\n\nA.\n\n# Other {#scope}\n\nB." - result = validate_document(_parse(body, "")) - assert [entry.identifier for entry in result.sections] == ["scope-2", "scope"] + result = validate(_parse(body, "")) + assert [entry.identifier for entry in result.index.sections] == ["scope-2", "scope"] assert "anchor-autogen-collision" in result.rules("warning") def test_two_generated_identifiers_collide_as_a_warning(): - result = validate_document(_parse("# Scope\n\nA.\n\n# Scope\n\nB.", "")) - assert [entry.identifier for entry in result.sections] == ["scope", "scope-2"] + result = validate(_parse("# Scope\n\nA.\n\n# Scope\n\nB.", "")) + assert [entry.identifier for entry in result.index.sections] == ["scope", "scope-2"] assert result.rules() == {"anchor-autogen-collision"} @@ -229,7 +236,7 @@ def test_a_reference_must_resolve_whenever_it_is_present(body, unsafe): def test_terms_and_attachments_are_checked_too(): frontmatter = _QUESTIONS + "attachments:\n - id: dpa\n title: DPA\n file: dpa.pdf\n when: vat\n" body = '# A {when=vat}\n\n"Fee" {{def: fee}} means the fee.\n\n# B\n\nPay the {{term: fee}} under {{attach: dpa}}.' - result = validate_document(_parse(body, frontmatter)) + result = validate(_parse(body, frontmatter)) messages = [d.message for d in result.diagnostics if d.rule == "condition-reference-unsafe"] assert len(messages) == 2 @@ -272,7 +279,7 @@ def test_a_condition_alone_makes_a_template(): def test_the_final_check_reports_conditions(): body = "# A {when=vat}\n\nText. {when=!vat}\n\n- item {#i when=vat}" - result = validate_document(_parse(body), final=True) + result = validate(_parse(body), final=True) rules = [d.rule for d in result.diagnostics] # the questions key and three conditions assert rules.count("template-construct-present") == 4 @@ -296,7 +303,7 @@ def test_a_question_may_be_used_in_a_fragment_the_validator_does_not_read(): def test_a_unit_that_can_never_appear_is_reported_once(): body = "# A {when=vat}\n\n- item {when=!vat} and {when=!vat}" - result = validate_document(_parse(body)) + result = validate(_parse(body)) assert [d.rule for d in result.diagnostics].count("condition-never-true") == 1 @@ -310,9 +317,9 @@ def test_a_unit_that_can_never_appear_is_reported_once(): def _with_attachment_definitions(body: str): - return validate_document( + return validate( _parse(body, _ALTERNATIVE_ATTACHMENTS), - import_attachment_definitions=lambda path: {"fee": f"Fee ({path})"}, + resolve=lambda path: _original(fee=f"Fee ({path})"), ) @@ -327,9 +334,9 @@ def test_a_term_from_a_conditional_attachment_is_checked_for_safety(): ) assert "condition-reference-unsafe" not in covered.rules() # one of the two is always present frontmatter = _QUESTIONS + "attachments:\n - id: s\n title: S\n file: s.lgd\n when: vat\n" - result = validate_document( + result = validate( _parse("# A\n\nSee {{attach: s}}. {when=vat}\n\nThe {{term: fee}}.", frontmatter), - import_attachment_definitions=lambda path: {"fee": "Fee"}, + resolve=lambda path: _original(fee="Fee"), ) assert "condition-reference-unsafe" in result.rules("error") @@ -344,10 +351,9 @@ def test_a_term_from_the_amended_original_is_always_present(): "amends:\n title: Original\n file: original.lgd\n" "attachments:\n - id: s\n title: S\n file: s.lgd\n when: vat\n" ) - result = validate_document( + result = validate( _parse("# A\n\nSee {{attach: s}}. {when=vat}", frontmatter), - import_definitions=lambda *_: {"fee": "Fee"}, - import_attachment_definitions=lambda path: {"fee": "Fee"}, + resolve=lambda path: _original(fee="Fee"), ) assert "def-duplicate-id" in result.rules("error") @@ -364,18 +370,18 @@ def test_a_preamble_list_item_carries_no_condition(): def test_a_term_redefined_under_a_condition_stays_defined_by_the_original(): frontmatter = _QUESTIONS + "amends:\n title: Original\n file: original.lgd\n" body = '# A\n\n"Services" {{def: services}} means x. {when=vat}\n\nWe provide {{term: services}}.' - result = validate_document(_parse(body, frontmatter), import_definitions=lambda *_: {"services": "Services"}) + result = validate(_parse(body, frontmatter), resolve=lambda path: _original(services="Services")) assert "condition-reference-unsafe" not in result.rules() def test_a_preamble_list_items_condition_is_explained(): - result = validate_document(_parse("- item {when=vat}\n\n# A\n\nText.")) + result = validate(_parse("- item {when=vat}\n\n# A\n\nText.")) [message] = [d.message for d in result.diagnostics if d.rule == "anchor-misplaced"] assert "only a paragraph may carry a condition" in message def test_alternative_attachments_are_unreferenced_once(): - result = validate_document(_parse("# A\n\nText.", _ALTERNATIVE_ATTACHMENTS)) + result = validate(_parse("# A\n\nText.", _ALTERNATIVE_ATTACHMENTS)) assert [d.rule for d in result.diagnostics].count("attachment-unreferenced") == 1 @@ -383,7 +389,7 @@ def test_a_comment_may_follow_a_heading_marker(): document = _parse("# Termination {when=vat} \n\nText.") [section] = document.sections assert (section.title, section.condition) == ("Termination ", "vat") - assert parse_document(serialize_document(document)).sections[0].condition == "vat" + assert parse(serialize_document(document)).sections[0].condition == "vat" def test_a_dotted_path_is_not_a_reference(): @@ -391,8 +397,8 @@ def test_a_dotted_path_is_not_a_reference(): def test_a_comment_in_a_heading_is_not_part_of_its_identifier(): - result = validate_document(_parse("# Title {when=vat} \n\nSee {{ref: title}}.")) - assert result.sections[0].identifier == "title" + result = validate(_parse("# Title {when=vat} \n\nSee {{ref: title}}.")) + assert result.index.sections[0].identifier == "title" assert "ref-broken" not in result.rules() @@ -405,8 +411,8 @@ def test_alternatives_share_a_number(): "# Disputes {#disputes when=forum:courts}\n\n## Venue\n\nText.\n\n" "# Disputes {#disputes when=forum:arbitration}\n\n## Seat\n\nText.\n\n# Notices\n\nText." ) - result = validate_document(_parse(body)) - assert [s.number for s in result.sections] == ["1", "1.1", "1", "1.1", "2"] + result = validate(_parse(body)) + assert [s.number for s in result.index.sections] == ["1", "1.1", "1", "1.1", "2"] def test_what_alternatives_contain_shares_numbers_after_a_skip(): @@ -416,8 +422,8 @@ def test_what_alternatives_contain_shares_numbers_after_a_skip(): "# Root\n\n## Choice {#choice when=forum:courts}\n\n### Real\n\n##### Deep One\n\n" "## Choice {#choice when=forum:arbitration}\n\n##### Deep Two\n\n## After\n" ) - result = validate_document(_parse(body)) - assert [s.number for s in result.sections] == ["1", "1.1", "1.1.1", "1.1.1.1.1", "1.1", "1.1.1.1.1", "1.2"] + result = validate(_parse(body)) + assert [s.number for s in result.index.sections] == ["1", "1.1", "1.1.1", "1.1.1.1.1", "1.1", "1.1.1.1.1", "1.2"] @pytest.mark.parametrize( @@ -486,7 +492,7 @@ def test_an_amendments_term_is_not_checked_when_its_original_is_unread(): def test_a_nested_items_presence_includes_the_items_it_is_nested_in(): body = "# A\n\n- parent {when=vat}\n - child {when=!vat}\n- other" - result = validate_document(_parse(body)) + result = validate(_parse(body)) messages = [d.message for d in result.diagnostics if d.rule == "condition-never-true"] assert len(messages) == 1 and "'{when=!vat}'" in messages[0] @@ -500,7 +506,7 @@ def test_every_item_a_nested_item_is_in_counts(): def test_an_item_nested_in_one_that_can_never_appear_is_not_reported_again(): body = "# A {when=vat}\n\n- a {when=!vat}\n - b {when=!vat}" - result = validate_document(_parse(body)) + result = validate(_parse(body)) assert [d.rule for d in result.diagnostics].count("condition-never-true") == 1 diff --git a/tests/test_identifiers.py b/tests/test_identifiers.py index 0d87b65..324e34b 100644 --- a/tests/test_identifiers.py +++ b/tests/test_identifiers.py @@ -3,8 +3,8 @@ import pytest -from legaldown.parser import parse_document -from legaldown.validator import validate_document +from legaldown.parser import parse +from legaldown.validator import validate from legaldown.validator.helpers import generate_identifier, slugify_identifier @@ -56,8 +56,8 @@ def test_a_slug_is_lossy_when_a_letter_or_digit_is_dropped(text, lossy): def _rules(body: str) -> list[str]: - document = parse_document(f"---\ntitle: Fixture\n---\n\n{body}\n") - return [d.rule for d in validate_document(document).diagnostics] + document = parse(f"---\ntitle: Fixture\n---\n\n{body}\n") + return [d.rule for d in validate(document).diagnostics] def test_a_heading_that_loses_letters_is_a_warning(): diff --git a/tests/test_positions.py b/tests/test_positions.py index 6ce2211..691edc8 100644 --- a/tests/test_positions.py +++ b/tests/test_positions.py @@ -6,11 +6,11 @@ import pytest -from legaldown import document_from_dict, document_to_dict, parse_document +from legaldown import document_from_dict, document_to_dict, parse from legaldown.cli import main from legaldown.models import Block from legaldown.parser import FrontmatterError -from legaldown.validator import validate_document +from legaldown.validator import validate _SIDES = """sides: - name: providers @@ -29,7 +29,7 @@ def _lines(source: str, rule: str) -> list[int | None]: - result = validate_document(parse_document(source, filename="t.lgd")) + result = validate(parse(source, filename="t.lgd")) return [d.line for d in result.diagnostics if d.rule == rule] @@ -73,7 +73,7 @@ def test_a_merged_key_is_where_it_is_written(): "title: T\nsides:\n - &base\n name: providers\n parties:\n - name: acme\n" " type: legal_entity\n - <<: *base\n name: clients\n" ) - document = parse_document(f"---\n{frontmatter}---\n\n# A\n") + document = parse(f"---\n{frontmatter}---\n\n# A\n") assert document.source_map.key("sides", 1, "parties", 0, "name") == 7 assert document.source_map.key("sides", 1, "name") == 10 assert _lines(f"---\n{frontmatter}---\n\n# A\n", "party-name-duplicate") == [7] @@ -85,9 +85,9 @@ def test_many_directives_in_one_block_are_found_quickly(): rows = "\n".join(f"| {{{{ref: bad-{n}}}}} | x |" for n in range(3000)) source = _HEAD + "| a | b |\n|---|---|\n" + rows + "\n\n" + " ".join(f"{{{{ref: p-{n}}}}}" for n in range(3000)) - document = parse_document(source) + document = parse(source) start = time.perf_counter() - lines = [d.line for d in validate_document(document).diagnostics if d.rule == "ref-broken"] + lines = [d.line for d in validate(document).diagnostics if d.rule == "ref-broken"] assert time.perf_counter() - start < 5 assert lines[:3] == [_BODY + 2, _BODY + 3, _BODY + 4] assert lines[-1] == _BODY + 3003 @@ -95,10 +95,10 @@ def test_many_directives_in_one_block_are_found_quickly(): def test_frontmatter_that_cannot_be_read_names_its_line(): with pytest.raises(FrontmatterError) as raised: - parse_document("---\ntitle: Fixture\n bad indent: [unclosed\n---\n\n# A\n") + parse("---\ntitle: Fixture\n bad indent: [unclosed\n---\n\n# A\n") assert raised.value.line == 3 with pytest.raises(FrontmatterError) as raised: - parse_document('---\ntitle: T\nsubtitle: "open\nlanguage: en\n---\n') + parse('---\ntitle: T\nsubtitle: "open\nlanguage: en\n---\n') assert raised.value.line == 3 # where the quote opens @@ -162,7 +162,7 @@ def test_a_document_without_frontmatter_counts_from_its_first_line(): def test_every_diagnostic_names_its_file(): - result = validate_document(parse_document(_HEAD + "{{ref: nope}}\n", filename="contract.lgd")) + result = validate(parse(_HEAD + "{{ref: nope}}\n", filename="contract.lgd")) assert {d.file for d in result.diagnostics} == {"contract.lgd"} @@ -170,22 +170,22 @@ def test_every_diagnostic_names_its_file(): def test_a_document_from_a_dict_has_no_lines(): - document = parse_document(_HEAD + "{{ref: nope}}\n") + document = parse(_HEAD + "{{ref: nope}}\n") rebuilt = document_from_dict(document_to_dict(document)) assert rebuilt == document and rebuilt.source_map is None assert "source_map" not in document_to_dict(document) - assert [d.line for d in validate_document(rebuilt).diagnostics] == [None] + assert [d.line for d in validate(rebuilt).diagnostics] == [None] def test_a_document_changed_after_parsing_names_no_line(): """Its source map describes the source as parsed: stale lines would point at the wrong text.""" - document = parse_document(_HEAD + "{{ref: nope}}\n\nText.\n") + document = parse(_HEAD + "{{ref: nope}}\n\nText.\n") document.sections[0].blocks.insert(0, Block(kind="paragraph", text="New.")) - assert [d.line for d in validate_document(document).diagnostics] == [None] - document = parse_document(_HEAD + "Text {{ref: nope}} and {{term: nope}}.\n") + assert [d.line for d in validate(document).diagnostics] == [None] + document = parse(_HEAD + "Text {{ref: nope}} and {{term: nope}}.\n") document.sections[0].blocks[0].suffix = " and {{term: other}}." - assert {d.line for d in validate_document(document).diagnostics} == {None} + assert {d.line for d in validate(document).diagnostics} == {None} # ── Output ───────────────────────────────────────────────────────── @@ -245,7 +245,7 @@ def test_each_reference_is_reported_at_its_own_line(seed): body, refs = _piece(r, n) real += [(target, len(lines) + index + 1) for target, index in refs] lines += [*body, ""] - result = validate_document(parse_document("\n".join(lines))) + result = validate(parse("\n".join(lines))) got = sorted( (int(d.message.split("'")[1].split("-")[1]), d.line) for d in result.diagnostics if d.rule == "ref-broken" ) diff --git a/tests/test_public_api.py b/tests/test_public_api.py index 18e2403..b1e16c4 100644 --- a/tests/test_public_api.py +++ b/tests/test_public_api.py @@ -1,8 +1,9 @@ -"""The public API (#26): what ``validate_document`` decides for a renderer — +"""The public API (#26): what ``validate`` decides for a renderer — the template decision and the markers that apply — and the helpers a renderer builds with, importable from ``legaldown`` and ``legaldown.validator``.""" from __future__ import annotations +import dataclasses import os from pathlib import Path @@ -15,11 +16,13 @@ ValidationResult, block_fragments, document_from_dict, + file_loader, is_drafting_note, is_template, list_fragments, - parse_document, - validate_document, + load, + parse, + validate, ) from legaldown.directives import lex from legaldown.validator.units import find_markers @@ -28,7 +31,7 @@ def _markers(body: str) -> list[PlacedMarker]: - return validate_document(parse_document(_FRONTMATTER + body)).placed_markers + return validate(parse(_FRONTMATTER + body)).index.placed_markers @pytest.mark.parametrize("module", [legaldown, legaldown.validator]) @@ -48,7 +51,7 @@ def test_the_renderers_names_are_public(): def test_a_result_built_directly_has_neither_decision(): result = ValidationResult() - assert (result.is_template, result.placed_markers) == (False, []) + assert (result.index.is_template, result.index.placed_markers) == (False, []) def test_a_marker_ending_a_paragraph_over_lines(): @@ -65,7 +68,7 @@ def test_a_list_items_marker_names_its_item(): markers = _markers(body) assert [(m.identifier, m.fragment, m.item) for m in markers] == [("a", 0, 0), ("b", 2, 1), ("c", 3, 2)] assert markers[2].condition == "x" - [block] = parse_document(_FRONTMATTER + body).sections[0].blocks + [block] = parse(_FRONTMATTER + body).sections[0].blocks assert [list_fragments(block)[m.fragment][2][-1] for m in markers] == [0, 1, 2] @@ -76,9 +79,9 @@ def test_an_include_only_paragraphs_identifier_does_not_apply(): def test_a_preamble_condition_applies_only_in_a_template(): assert _markers("Intro {when=a}\n\n# A\n\nText.\n") == [] - result = validate_document(parse_document(_FRONTMATTER + "Intro {when=a}\n\n# A {when=b}\n\nText.\n")) - assert result.is_template - assert [(m.section, m.condition) for m in result.placed_markers] == [(None, "a")] + result = validate(parse(_FRONTMATTER + "Intro {when=a}\n\n# A {when=b}\n\nText.\n")) + assert result.index.is_template + assert [(m.section, m.condition) for m in result.index.placed_markers] == [(None, "a")] def test_the_offset_is_the_markers_not_a_copy_in_a_comment(): @@ -99,17 +102,17 @@ def test_a_marker_out_of_place_is_not_placed(body): def test_a_document_built_in_code_has_no_lines(): document = document_from_dict({"sections": [{"title": "A", "blocks": [{"kind": "paragraph", "text": "Text {#t}"}]}]}) - [marker] = validate_document(document).placed_markers + [marker] = validate(document).index.placed_markers assert (marker.identifier, marker.line) == ("t", None) def test_is_template_on_its_own(): - assert not is_template(parse_document(_FRONTMATTER + "# A\n\nText.\n")) - assert is_template(parse_document(_FRONTMATTER + "# A\n\nPick {{choose: q}}.\n")) + assert not is_template(parse(_FRONTMATTER + "# A\n\nText.\n")) + assert is_template(parse(_FRONTMATTER + "# A\n\nPick {{choose: q}}.\n")) def test_is_drafting_note(): - [note, quote] = parse_document(_FRONTMATTER + "# A\n\n> [!drafting]\n> Check.\n\n> Plain.\n").sections[0].blocks + [note, quote] = parse(_FRONTMATTER + "# A\n\n> [!drafting]\n> Check.\n\n> Plain.\n").sections[0].blocks assert is_drafting_note(note) and not is_drafting_note(quote) @@ -127,19 +130,19 @@ def test_the_decisions_are_the_validators_own(path: Path): ``is_template``, the placed markers are the validator's placed findings, each where it says, at most one in a fragment.""" try: - document = parse_document(path.read_text(encoding="utf-8")) + document = parse(path.read_text(encoding="utf-8")) except legaldown.FrontmatterError: pytest.skip("unreadable frontmatter") - result = validate_document(document) - assert result.is_template == is_template(document) - found = [f for f in find_markers(document, lex) if f.marker is not None and f.placed(result.is_template)] - assert [(m.section, m.block, m.fragment, m.offset, m.source) for m in result.placed_markers] == [ + result = validate(document) + assert result.index.is_template == is_template(document) + found = [f for f in find_markers(document, lex) if f.marker is not None and f.placed(result.index.is_template)] + assert [(m.section, m.block, m.fragment, m.offset, m.source) for m in result.index.placed_markers] == [ (f.section, f.block, f.fragment, f.offset, f.source) for f in found ] - places = [(m.section, m.block, m.fragment) for m in result.placed_markers] + places = [(m.section, m.block, m.fragment) for m in result.index.placed_markers] assert len(places) == len(set(places)) lines = path.read_text(encoding="utf-8").split("\n") - for marker in result.placed_markers: + for marker in result.index.placed_markers: blocks = document.preamble if marker.section is None else document.sections[marker.section].blocks text = block_fragments(blocks[marker.block])[marker.fragment][0] assert text[marker.offset:marker.offset + len(marker.source)] == marker.source @@ -148,7 +151,7 @@ def test_the_decisions_are_the_validators_own(path: Path): def test_a_marker_after_a_lifted_reference_is_in_the_suffix(): [marker] = _markers("# A\n\nSee {{ref: a}} below. {#t}\n\n# B {#a}\n\nText.\n") - block = parse_document(_FRONTMATTER + "# A\n\nSee {{ref: a}} below. {#t}\n\n# B {#a}\n\nText.\n").sections[0].blocks[0] + block = parse(_FRONTMATTER + "# A\n\nSee {{ref: a}} below. {#t}\n\n# B {#a}\n\nText.\n").sections[0].blocks[0] assert (block.kind, marker.field) == ("ref", "suffix") assert block.suffix[marker.offset:] == "{#t}" @@ -175,16 +178,16 @@ def test_a_list_is_walked_once(monkeypatch): def test_a_document_changed_since_parsing_has_no_lines(): - document = parse_document(_FRONTMATTER + "# A\n\nText {#t}\n\n- a {#b}\n") + document = parse(_FRONTMATTER + "# A\n\nText {#t}\n\n- a {#b}\n") document.sections[0].blocks.insert(0, document.sections[0].blocks[0]) - assert [m.line for m in validate_document(document).placed_markers] == [None, None, None] + assert [m.line for m in validate(document).index.placed_markers] == [None, None, None] def test_a_list_of_string_items_built_in_code(): document = document_from_dict({"sections": [{"title": "A", "blocks": [ {"kind": "unordered_list", "items": ["a {#x}", "b\n\n - c {#y}"]}, ]}]}) - assert [(m.identifier, m.item) for m in validate_document(document).placed_markers] == [("x", 0), ("y", 2)] + assert [(m.identifier, m.item) for m in validate(document).index.placed_markers] == [("x", 0), ("y", 2)] def test_two_lists_in_a_section_number_their_own_items(): @@ -199,7 +202,7 @@ def test_two_lists_in_a_section_number_their_own_items(): def test_a_marker_after_a_lifted_term_is_in_the_suffix(): body = '"Thing" {{def: thing}} means a thing.\n\n# A\n\nUse the {{term: thing}} well. {#t}\n' [marker] = _markers(body) - block = parse_document(_FRONTMATTER + body).sections[0].blocks[0] + block = parse(_FRONTMATTER + body).sections[0].blocks[0] assert (block.kind, marker.field) == ("term", "suffix") assert block.suffix[marker.offset:] == "{#t}" @@ -212,8 +215,354 @@ def test_only_a_quote_block_is_a_drafting_note(): def test_fragments_have_names(): - [block] = parse_document(_FRONTMATTER + "# A\n\n- a {#a}\n - b\n").sections[0].blocks + [block] = parse(_FRONTMATTER + "# A\n\n- a {#a}\n - b\n").sections[0].blocks first = list_fragments(block)[0] assert (first.text, first.anchor, first.items) == ("a {#a}", True, (0,)) text, anchor = block_fragments(block)[1] assert (text, anchor) == ("b", True) + + +# load / parse ---------------------------------------------------------------- + +_SOURCE = _FRONTMATTER + "# A {#a}\n\nText.\n" + + +def test_load_reads_a_file_and_names_it(tmp_path): + file = tmp_path / "contract.lgd" + file.write_text(_SOURCE, encoding="utf-8") + document = load(file) + assert (document.filename, document.path) == ("contract.lgd", file.resolve()) + assert document.sections[0].title == "A" + + +def test_load_takes_a_string_path_and_a_relative_one(tmp_path, monkeypatch): + (tmp_path / "contract.lgd").write_text(_SOURCE, encoding="utf-8") + monkeypatch.chdir(tmp_path) + assert load("contract.lgd").path == (tmp_path / "contract.lgd").resolve() + assert load("contract.lgd").path.is_absolute() + + +@pytest.mark.parametrize("ending", ["\n", "\r\n", "\r"]) +def test_load_reads_the_same_whatever_the_line_endings(tmp_path, ending): + file = tmp_path / "x.lgd" + file.write_bytes(_SOURCE.replace("\n", ending).encode("utf-8")) + assert load(file) == parse(_SOURCE, filename="x.lgd") + + +def test_load_drops_a_byte_order_mark_and_keeps_utf8(tmp_path): + source = _FRONTMATTER + "# Čl. 1\n\nPlatba v €.\n" + file = tmp_path / "x.lgd" + file.write_bytes(b"\xef\xbb\xbf" + source.encode("utf-8")) + assert load(file) == parse(source, filename="x.lgd") + + +def test_load_has_the_source_map_of_a_parsed_document(tmp_path): + file = tmp_path / "x.lgd" + file.write_text(_FRONTMATTER + "# A\n\nSee {{ref: nowhere}}.\n", encoding="utf-8") + [diagnostic] = [d for d in validate(load(file)).diagnostics if d.rule == "ref-broken"] + assert (diagnostic.line, diagnostic.file) == (7, "x.lgd") + + +def test_load_fails_in_the_ways_the_documentation_says(tmp_path): + with pytest.raises(FileNotFoundError): + load(tmp_path / "nope.lgd") + bad = tmp_path / "bad.lgd" + bad.write_bytes(b"\xff\xfe") + with pytest.raises(UnicodeDecodeError): + load(bad) + bad.write_text("---\ntitle: [unclosed\n---\n", encoding="utf-8") + with pytest.raises(legaldown.FrontmatterError): + load(bad) + + +def test_where_a_document_lives_is_not_part_of_what_it_says(tmp_path): + file = tmp_path / "x.lgd" + file.write_text(_SOURCE, encoding="utf-8") + assert load(file).path is not None + assert parse(_SOURCE).path is None + assert load(file).path != parse(_SOURCE).path + assert load(file) == parse(_SOURCE, filename="x.lgd") + assert "path" not in legaldown.document_to_dict(load(file)) + + +def test_parse_document_is_a_deprecated_alias_of_parse(): + with pytest.warns(DeprecationWarning, match=r"removed in 0\.5\.0") as record: + document = legaldown.parse_document(_SOURCE, filename="x.lgd") + assert document == parse(_SOURCE, filename="x.lgd") + assert record[0].filename == __file__ # the caller is named, not the shim + + +def test_a_symbolic_link_is_named_as_it_was_given(tmp_path): + target = tmp_path / "v3-final.lgd" + target.write_text(_SOURCE, encoding="utf-8") + link = tmp_path / "current.lgd" + try: + link.symlink_to(target) + except OSError: + pytest.skip("symbolic links are not available") + document = load(link) + assert (document.filename, document.path) == ("current.lgd", link) + + +def test_dot_dot_after_a_symbolic_link_goes_where_the_system_goes(tmp_path, monkeypatch): + sub = tmp_path / "data" / "sub" + sub.mkdir(parents=True) + (tmp_path / "data" / "contract.lgd").write_text(_FRONTMATTER + "# Real\n", encoding="utf-8") + (tmp_path / "contract.lgd").write_text(_FRONTMATTER + "# Wrong\n", encoding="utf-8") + try: + (tmp_path / "link").symlink_to(sub) + except OSError: + pytest.skip("symbolic links are not available") + monkeypatch.chdir(tmp_path) + assert load("link/../contract.lgd").sections[0].title == "Real" + + +def test_document_keeps_source_map_in_its_place(): + fields = [f.name for f in dataclasses.fields(legaldown.Document)] + assert fields.index("source_map") == 4 # positional callers: unchanged by path + + +# validate: what it reads, what it returns ------------------------------------- + +_AMENDMENT = "---\ntitle: A\namends:\n title: Original\n file: {file}\n---\n\n# S\n\nUses {{{{term: services}}}}.\n" +_ORIGINAL = '---\ntitle: O\n---\n\n# S\n\n"Services" {{def: services}} means x.\n' +_EMPTY_ORIGINAL = "---\ntitle: O\n---\n\n# S\n\nText.\n" + + +def _amendment(tmp_path, original: str | None, *, file: str = "original.lgd", name: str = "amendment.lgd"): + if original is not None: + (tmp_path / "original.lgd").write_text(original, encoding="utf-8") + path = tmp_path / name + path.write_text(_AMENDMENT.format(file=file), encoding="utf-8") + return load(path) + + +def test_validate_reads_the_amended_original_from_beside_the_document(tmp_path): + assert "amend-term-undefined" not in validate(_amendment(tmp_path, _ORIGINAL)).rules() + assert "amend-term-undefined" in validate(_amendment(tmp_path, _EMPTY_ORIGINAL)).rules("error") + + +def test_a_document_without_a_path_has_no_original_to_read(): + result = validate(parse(_AMENDMENT.format(file="original.lgd"))) + assert "amend-term-unresolvable" in result.rules() + + +@pytest.mark.parametrize("original", [None, "---\ntitle: [unclosed\n---\n", b"\xff\xfe"]) +def test_an_original_that_cannot_be_read_is_not_a_finding_of_its_own(tmp_path, original): + unreadable = validate(parse(_AMENDMENT.format(file="original.lgd"))) + if isinstance(original, bytes): + (tmp_path / "original.lgd").write_bytes(original) + document = _amendment(tmp_path, None) + else: + document = _amendment(tmp_path, original) + assert validate(document).rules() == unreadable.rules() + + +def test_an_original_outside_the_documents_directory_is_not_read(tmp_path): + inner = tmp_path / "inner" + inner.mkdir() + (tmp_path / "original.lgd").write_text(_ORIGINAL, encoding="utf-8") + document = _amendment(inner, None, file="../original.lgd") + assert "amend-term-unresolvable" in validate(document).rules() + + +def test_only_the_originals_own_definitions_are_read(tmp_path): + (tmp_path / "base.lgd").write_text(_ORIGINAL, encoding="utf-8") + original = "---\ntitle: O\namends:\n title: Base\n file: base.lgd\n---\n\n# S\n\nText.\n" + assert "amend-term-undefined" in validate(_amendment(tmp_path, original)).rules("error") + + +def test_resolve_stands_in_for_the_filesystem(tmp_path): + document = _amendment(tmp_path, _EMPTY_ORIGINAL) + asked: list[str] = [] + + def resolve(path: str) -> str: + asked.append(path) + return _ORIGINAL + + assert "amend-term-undefined" not in validate(document, resolve=resolve).rules() + assert asked == ["original.lgd"] + assert "amend-term-undefined" not in validate(parse(_AMENDMENT.format(file="o.lgd")), resolve=resolve).rules() + + +def test_the_importer_callbacks_still_work_and_say_they_are_deprecated(): + document = parse(_AMENDMENT.format(file="original.lgd")) + with pytest.warns(DeprecationWarning, match=r"removed in 0\.5\.0"): + result = validate(document, import_definitions=lambda *_: {"services": "Services"}) + assert "amend-term-undefined" not in result.rules() and "amend-term-unresolvable" not in result.rules() + with pytest.warns(DeprecationWarning, match=r"removed in 0\.5\.0"): + validate(document, import_attachment_definitions=lambda _file: {}) + + +def test_validate_document_is_a_deprecated_alias_of_validate(): + document = parse(_SOURCE) + with pytest.warns(DeprecationWarning, match=r"removed in 0\.5\.0") as record: + result = legaldown.validate_document(document, final=True) + assert result == validate(document, final=True) + assert record[0].filename == __file__ + + +def test_the_messages_by_severity_are_the_diagnostics_own(): + result = validate(parse(_FRONTMATTER + "# A\n\nSee {{ref: nowhere}}.\n")) + assert result.errors == [d.message for d in result.diagnostics if d.level == "error"] != [] + assert result.warnings == [d.message for d in result.diagnostics if d.level == "warning"] + assert result.infos == [d.message for d in result.diagnostics if d.level == "info"] + assert not result.is_valid + result.diagnostics.clear() + assert (result.errors, result.is_valid) == ([], True) # nothing kept apart from them + + +def test_a_result_is_what_was_found_not_the_validators_recorder(): + result = validate(parse(_SOURCE)) + assert type(result) is ValidationResult + assert not any(hasattr(result, name) for name in ("error", "warning", "info", "at", "used_terms")) + + +# file_loader ----------------------------------------------------------------- + + +def test_file_loader_reads_beside_and_below_the_base_as_written(tmp_path): + (tmp_path / "sub").mkdir() + (tmp_path / "a.lgd").write_bytes(b"one\r\ntwo\r\n") + (tmp_path / "sub" / "b.lgd").write_text("b", encoding="utf-8") + load_file = file_loader(tmp_path) + assert load_file("a.lgd") == "one\r\ntwo\r\n" # no line-break translation + assert load_file("sub/b.lgd") == "b" + assert load_file("sub/../a.lgd") == "one\r\ntwo\r\n" + + +def test_file_loader_reads_nothing_it_may_not(tmp_path): + base = tmp_path / "base" + base.mkdir() + (tmp_path / "outside.lgd").write_text("x", encoding="utf-8") + (base / "latin.lgd").write_bytes(b"caf\xe9") + load_file = file_loader(base) + assert load_file("../outside.lgd") is None + assert load_file(str(tmp_path / "outside.lgd")) is None + assert load_file("missing.lgd") is None + assert load_file("") is None + assert load_file(".") is None # a directory + assert load_file("a\0b") is None + assert load_file("latin.lgd") is None # not UTF-8 + + +def test_a_loaded_document_can_be_validated_without_reading_anything(tmp_path): + document = _amendment(tmp_path, _ORIGINAL) + assert "amend-term-undefined" not in validate(document).rules() + assert "amend-term-unresolvable" in validate(document, resolve=lambda path: None).rules() + + +def test_a_document_that_names_itself_as_its_original_reads_nothing(tmp_path): + path = tmp_path / "amendment.lgd" + path.write_text( + '---\ntitle: A\namends:\n title: A\n file: ./amendment.lgd\n---\n\n# S\n\n"Services" {{def: services}} means x.\n', + encoding="utf-8", + ) + for options in ({}, {"resolve": file_loader(tmp_path)}): # by default, and through any resolver + rules = validate(load(path), **options).rules() + assert "amend-def-override" not in rules + + +def test_one_importer_does_not_start_reading_the_other_files_from_disk(tmp_path): + document = _amendment(tmp_path, _ORIGINAL) + with pytest.warns(DeprecationWarning): + # An importer for the attachments only: the amended original is still not read, as it never was. + result = validate(document, import_attachment_definitions=lambda _file: {}) + assert "amend-term-unresolvable" in result.rules() + + +@pytest.mark.parametrize( + ("written", "asked"), + [("original.lgd", ["original.lgd"]), ("./sub/../original.lgd", ["original.lgd"]), + ("../original.lgd", []), ("/original.lgd", []), ("sub/../../original.lgd", [])], +) +def test_a_resolver_is_asked_for_a_normalized_path_within_the_directory(written, asked): + document = parse(_AMENDMENT.format(file=written)) + seen: list[str] = [] + validate(document, resolve=lambda path: seen.append(path)) + assert seen == asked + + +# DocumentIndex --------------------------------------------------------------- + + +def test_the_analysis_is_in_the_index_and_the_result_holds_findings_and_index(): + result = validate(parse(_FRONTMATTER + "# A {#a}\n\nSee {{ref: a}}. {{date: 2026-06-01}} {{money: 5, currency=EUR}}\n")) + assert [f.name for f in dataclasses.fields(result)] == ["diagnostics", "index"] + assert isinstance(result.index, legaldown.DocumentIndex) and isinstance(result.index.values, legaldown.InlineValues) + assert [e.identifier for e in result.index.sections] == ["a"] and "a" in result.index.section_lookup + assert (result.index.values.dates, result.index.values.money) == (["2026-06-01"], [("5", "EUR")]) + assert result.index.is_template is False and result.index.placed_markers == [] + + +@pytest.mark.parametrize( + "old", + ["sections", "section_lookup", "definition_lookup", "party_lookup", "side_lookup", "attachment_lookup", + "inline_dates", "inline_money", "inline_durations", "inline_fields", "inline_placeholders", + "is_template", "placed_markers"], +) +def test_the_analysis_is_not_on_the_result_any_more(old): + assert not hasattr(validate(parse(_SOURCE)), old) + + +def test_each_validation_has_an_index_of_its_own(): + first, second = validate(parse(_SOURCE)), validate(parse(_SOURCE)) + assert first.index == second.index and first.index is not second.index + assert first.index.values is not second.index.values + + +def test_a_result_is_built_by_keyword_so_an_old_positional_call_fails_loudly(): + with pytest.raises(TypeError): + ValidationResult([], []) + assert ValidationResult(diagnostics=[], index=legaldown.DocumentIndex()).is_valid + + +# relative_path and the paths a loader is asked for ----------------------------- + + +@pytest.mark.parametrize( + ("path", "relative"), + [ + ("a.lgd", "a.lgd"), ("./a.lgd", "a.lgd"), ("sub/../a.lgd", "a.lgd"), ("sub//b.lgd", "sub/b.lgd"), + ("../a.lgd", None), ("sub/../../a.lgd", None), ("..", None), (".", None), ("", None), + ("/etc/passwd", None), ("C:/x.lgd", None), ("C:x.lgd", None), ("c:\\x.lgd", None), + ("..\\secret.lgd", None), ("sub\\a.lgd", None), ("a\0b.lgd", None), + ], +) +def test_a_path_a_document_names_is_within_its_directory_or_none(path, relative): + from legaldown.files import relative_path + + assert relative_path(path) == relative + + +@pytest.mark.parametrize("hostile", ["../../secret.lgd", "/etc/passwd.lgd", "..\\secret.lgd", "C:/x.lgd", "C:x.lgd"]) +def test_a_loader_is_never_asked_for_a_path_outside_the_directory(hostile): + asked: list[str] = [] + + def resolve(path: str) -> str | None: + asked.append(path) + return None + + validate(parse(_AMENDMENT.format(file=hostile)), resolve=resolve) + text = f"---\ntitle: T\n---\n\n# A {{#a}}\n\n{{{{include: {hostile}}}}}\n" + template = legaldown.parse_template(text, resolve=resolve) + assert "include-file-missing" in [d.rule for d in template.problems] + with pytest.warns(DeprecationWarning): + assert not legaldown.assemble(text, {}, load_file=resolve).ok + assert asked == [] + + +def test_a_symbolic_link_inside_the_directory_to_a_file_outside_it_is_not_read(tmp_path): + base = tmp_path / "base" + base.mkdir() + (tmp_path / "outside.lgd").write_text("secret", encoding="utf-8") + try: + (base / "link.lgd").symlink_to(tmp_path / "outside.lgd") + (base / "dirlink").symlink_to(tmp_path) + except OSError: + pytest.skip("symbolic links are not available") + load_file = file_loader(base) + assert load_file("link.lgd") is None and load_file("dirlink/outside.lgd") is None + (base / "inside.lgd").write_text("fine", encoding="utf-8") + (base / "alias.lgd").symlink_to(base / "inside.lgd") + assert load_file("alias.lgd") == "fine" # a link that stays within is read diff --git a/tests/test_spec_alignment.py b/tests/test_spec_alignment.py index 2f2c64b..fca7dd5 100644 --- a/tests/test_spec_alignment.py +++ b/tests/test_spec_alignment.py @@ -25,8 +25,8 @@ from legaldown.directives import format_value, lex from legaldown.markers import Marker, split_heading from legaldown.models import CustomField, empty_document, list_items -from legaldown.parser import collect_source_directives, parse_document -from legaldown.validator import validate_document +from legaldown.parser import collect_source_directives, parse +from legaldown.validator import validate from legaldown.validator.helpers import generate_identifier _FRONTMATTER = """--- @@ -65,7 +65,7 @@ def _shape(block: Block): def _validate(body: str): - return validate_document(parse_document(_FRONTMATTER + body, filename="t.lgd")) + return validate(parse(_FRONTMATTER + body, filename="t.lgd")) # ── Faithful parsing ────────────────────────────────────────────── @@ -103,12 +103,12 @@ def test_parser_does_not_silently_correct_party_metadata(): Text. """ - document = parse_document(source, filename="t.lgd") + document = parse(source, filename="t.lgd") side = document.metadata.sides[0] assert side.name == "Providing Party" # preserved verbatim assert side.parties[0].type == "company" # not mapped onto legal_entity - rules = validate_document(document).rules("error") + rules = validate(document).rules("error") assert "side-party-name-format" in rules # the display name assert "party-type-invalid" in rules # the unknown type @@ -133,9 +133,9 @@ def test_a_party_omitting_the_required_type_is_reported(): Text. """ - document = parse_document(source, filename="t.lgd") + document = parse(source, filename="t.lgd") assert document.metadata.sides[0].parties[0].type == "" - assert "party-type-invalid" in validate_document(document).rules("error") + assert "party-type-invalid" in validate(document).rules("error") # ── §11.4 recognition contexts ──────────────────────────────────── @@ -180,7 +180,7 @@ def test_unknown_parameter_is_a_warning_and_does_not_suppress_checks(body, rule) def test_unknown_parameter_alone_leaves_the_document_valid(): result = _validate("On {{date: 2026-01-01, colour=red}}.") assert result.is_valid - assert result.inline_dates == ["2026-01-01"] + assert result.index.values.dates == ["2026-01-01"] @pytest.mark.parametrize( @@ -205,8 +205,8 @@ def test_parameters_in_any_order_are_valid(): "under {{field: A-1, note=Ref, type=case-id}}." ) assert result.diagnostics == [] - assert ("5", "USD") in result.inline_money - assert ("A-1", "case-id") in result.inline_fields + assert ("5", "USD") in result.index.values.money + assert ("A-1", "case-id") in result.index.values.fields def test_quoted_values_are_decoded(): @@ -217,9 +217,9 @@ def test_quoted_values_are_decoded(): 'and {{field: "a}}b", type=code}}.' ) assert result.diagnostics == [] - assert ("Smith, Jones v. Doe", "case-name") in result.inline_fields - assert ("a}}b", "code") in result.inline_fields - assert ("5", "USD") in result.inline_money + assert ("Smith, Jones v. Doe", "case-name") in result.index.values.fields + assert ("a}}b", "code") in result.index.values.fields + assert ("5", "USD") in result.index.values.money def test_quoted_values_are_still_checked(): @@ -256,22 +256,22 @@ def test_parameters_on_ref_and_def_are_reported(): ) result = _validate(source) assert [d.rule for d in result.diagnostics] == ["directive-unknown-param"] * 2 - assert result.definition_lookup["foo"] == "Foo" - assert "{{ref: terms, format=long}}" in serialize_document(parse_document(_FRONTMATTER + source)) + assert result.index.definition_lookup["foo"] == "Foo" + assert "{{ref: terms, format=long}}" in serialize_document(parse(_FRONTMATTER + source)) def test_directive_in_code_span_is_not_lifted_or_checked(): """§11.4: the parser must not lift a code-span {{ref:}} into a ref block.""" - document = parse_document(_FRONTMATTER + "Write `{{ref: nope}}` or `{{term: nope}}`.") + document = parse(_FRONTMATTER + "Write `{{ref: nope}}` or `{{term: nope}}`.") assert document.sections[0].blocks[0].kind == "paragraph" - assert validate_document(document).diagnostics == [] + assert validate(document).diagnostics == [] def test_lifted_term_label_round_trips(): source = _FRONTMATTER + '"Svc" {{def: svc}} x.\n\nSee {{term: svc, label="Services, as amended"}}.\n' - block = parse_document(source).sections[0].blocks[1] + block = parse(source).sections[0].blocks[1] assert (block.kind, block.target, block.label) == ("term", "svc", "Services, as amended") - assert '{{term: svc, label="Services, as amended"}}' in serialize_document(parse_document(source)) + assert '{{term: svc, label="Services, as amended"}}' in serialize_document(parse(source)) def test_placeholder_currency_is_defined_only_for_money(): @@ -285,7 +285,7 @@ def test_frontmatter_placeholder_with_unknown_parameter_is_checked(): source = _FRONTMATTER.replace( "title: Fixture", 'title: "{{placeholder: t, type=percentage, colour=red}}"' ) - result = validate_document(parse_document(source + "Text.\n", filename="t.lgd")) + result = validate(parse(source + "Text.\n", filename="t.lgd")) assert "placeholder-type-invalid" in result.rules("error") assert "directive-unknown-param" in result.rules("warning") @@ -323,10 +323,10 @@ def test_other_values_are_not_curly_quote_warnings(text): def test_a_curly_quoted_reference_stays_paragraph_text(): """A lifted {{ref:}} or {{term:}} is not lexed again, so a directive the validator warns about is not lifted.""" - document = parse_document(_FRONTMATTER + "See {{ref: “terms”}}.\n") + document = parse(_FRONTMATTER + "See {{ref: “terms”}}.\n") assert document.sections[0].blocks[0].kind == "paragraph" - assert parse_document(serialize_document(document)).sections == document.sections - assert "value-curly-quote" in validate_document(document).rules("warning") + assert parse(serialize_document(document)).sections == document.sections + assert "value-curly-quote" in validate(document).rules("warning") @pytest.mark.parametrize( @@ -334,15 +334,15 @@ def test_a_curly_quoted_reference_stays_paragraph_text(): ['"Fee" {{def: fee}} x.\n\nSee {{term: fee, label="“Curly”"}}.', 'See {{ref: "“terms”"}}.', '"Fee" {{def: "«fee»"}} x.'], ) def test_a_quoted_curly_value_stays_quoted_through_a_round_trip(text): - document = parse_document(_FRONTMATTER + text + "\n") - reparsed = parse_document(serialize_document(document)) + document = parse(_FRONTMATTER + text + "\n") + reparsed = parse(serialize_document(document)) assert reparsed.sections == document.sections - assert "value-curly-quote" not in validate_document(reparsed).rules() + assert "value-curly-quote" not in validate(reparsed).rules() def test_a_curly_quote_in_a_frontmatter_placeholder_is_a_warning(): source = _FRONTMATTER.replace("title: Fixture", "title: '{{placeholder: t, note=“x”}}'") - assert "value-curly-quote" in validate_document(parse_document(source + "Text.\n")).rules("warning") + assert "value-curly-quote" in validate(parse(source + "Text.\n")).rules("warning") def test_the_lexer_records_which_values_were_unquoted(): @@ -351,7 +351,7 @@ def test_the_lexer_records_which_values_were_unquoted(): def test_collect_source_directives_sees_every_parameter_shape(): - document = parse_document( + document = parse( _FRONTMATTER + "See {{ref: a, colour=red}} and {{term: b, label=\"x, y\"}}.\n" ) assert collect_source_directives(document) == ({"a"}, {"b"}) @@ -359,7 +359,7 @@ def test_collect_source_directives_sees_every_parameter_shape(): def test_explicitly_empty_label_round_trips(): source = _FRONTMATTER + '"X" {{def: x}} y.\n\nSee {{term: x, label=""}}.\n' - assert '{{term: x, label=""}}' in serialize_document(parse_document(source)) + assert '{{term: x, label=""}}' in serialize_document(parse(source)) def test_malformed_directive_with_unknown_name_is_malformed(): @@ -378,7 +378,6 @@ def test_missing_ref_and_term_targets_are_reported_as_missing(): messages = {d.rule: d.message for d in result.diagnostics} assert "has no target" in messages["ref-broken"] assert "has no target" in messages["term-undefined"] - assert "" not in result.used_terms @pytest.mark.parametrize( @@ -415,26 +414,26 @@ def test_definition_in_code_span_or_comment_is_not_registered(): def test_def_opener_with_inner_whitespace_is_not_a_definition(): """§11.2: no whitespace between {{ and the directive name.""" result = _validate('"Foo" {{ def: foo}} means x. Use {{term: foo}}.') - assert "foo" not in result.definition_lookup + assert "foo" not in result.index.definition_lookup def test_defined_term_is_the_nearest_quoted_span(): """§7.2: scan back from the closing mark to the nearest opening mark.""" result = _validate('Between "A" and "B" {{def: b}} use {{term: b}}.') - assert result.definition_lookup["b"] == "B" + assert result.index.definition_lookup["b"] == "B" def test_quoted_value_may_contain_backticks(): """§11.3: once a directive opens, its value is lexed as written.""" result = _validate('Case {{field: "a`b", type=code}} and `c` here.') assert result.diagnostics == [] - assert ("a`b", "code") in result.inline_fields + assert ("a`b", "code") in result.index.values.fields def test_lifted_values_are_kept_as_written(): source = _FRONTMATTER + '"X" {{def: x}} y.\n\nSee {{term: x, label="a `b` c"}}.\n' - assert parse_document(source).sections[0].blocks[1].label == "a `b` c" - reparsed = parse_document(serialize_document(parse_document(source))) + assert parse(source).sections[0].blocks[1].label == "a `b` c" + reparsed = parse(serialize_document(parse(source))) assert reparsed.sections[0].blocks[1].label == "a `b` c" @@ -462,7 +461,7 @@ def test_malformed_directive_ends_at_the_next_opener(body): def test_definition_with_other_quotation_marks_round_trips(paragraph): """The serializer writes straight quotes, so other delimiters stay text.""" source = _FRONTMATTER + paragraph + "\n" - assert paragraph in serialize_document(parse_document(source)) + assert paragraph in serialize_document(parse(source)) def test_ref_inside_a_defined_term_is_not_lifted(): @@ -505,14 +504,14 @@ def test_unclosed_directive_does_not_swallow_the_next(): @pytest.mark.parametrize("paragraph", ['"" {{def: foo}} means x.', '" Foo " {{def: foo}} means x.']) def test_definition_the_block_cannot_hold_stays_text(paragraph): source = _FRONTMATTER + paragraph + "\n" - assert paragraph in serialize_document(parse_document(source)) + assert paragraph in serialize_document(parse(source)) def test_escaped_placeholder_in_metadata_is_not_a_placeholder(): source = _FRONTMATTER.replace( "title: Fixture", 'title: Fixture\neffective_date: "\\\\{{placeholder: d}} soon"' ) - result = validate_document(parse_document(source + "Text.\n")) + result = validate(parse(source + "Text.\n")) assert "metadata-date-invalid" in result.rules("error") @@ -620,7 +619,7 @@ def test_format_value_is_the_inverse_of_the_lexer(value): def test_side_directive_resolves_and_reports_unknown(): ok = _validate("The {{side: clients}} shall pay.") assert "side-unknown" not in ok.rules() - assert ok.side_lookup["clients"] == "Clients" + assert ok.index.side_lookup["clients"] == "Clients" bad = _validate("The {{side: nobody}} shall pay.") assert "side-unknown" in bad.rules("error") @@ -629,8 +628,8 @@ def test_side_directive_resolves_and_reports_unknown(): def test_duration_unit_m_is_rejected_with_hint(): # Written directly in a directive the migration does not reach (already # canonical spelling context), the bare unit M is an error. - result = validate_document( - parse_document( + result = validate( + parse( _FRONTMATTER + "Within {{duration: 5, unit=Q}}.", filename="t.lgd" ) ) @@ -649,7 +648,7 @@ def test_ref_to_attachment_id_suggests_attach(): "document_type: contract\nattachments:\n" " - id: schedule-a\n title: Schedule A\n file: schedule-a.pdf", ) - result = validate_document(parse_document(source + body, filename="t.lgd")) + result = validate(parse(source + body, filename="t.lgd")) assert "ref-targets-attachment" in result.rules("error") @@ -677,7 +676,7 @@ def _numbers(headings: str) -> list[str]: """The section numbers of a document with *headings*; one that opens below level 1 is also a heading-skip, numbered all the same.""" source = _FRONTMATTER.replace("# Terms {#terms}\n\n", "") + headings - return [entry.number for entry in validate_document(parse_document(source)).sections] + return [entry.number for entry in validate(parse(source)).index.sections] @pytest.mark.parametrize( @@ -697,8 +696,8 @@ def test_a_skipped_level_counts_as_its_first_so_numbers_stay_unique(headings, nu def test_a_fragment_starting_at_level_two_is_numbered_from_one(): """An attachment or include file has neither frontmatter nor a # heading (§12, §13.8).""" - result = validate_document(parse_document("## A\n\n## B\n\n### B1\n\n## C\n")) - assert [entry.number for entry in result.sections] == ["1", "2", "2.1", "3"] + result = validate(parse("## A\n\n## B\n\n### B1\n\n## C\n")) + assert [entry.number for entry in result.index.sections] == ["1", "2", "2.1", "3"] assert "heading-skip" not in result.rules() @@ -709,7 +708,7 @@ def test_a_fragment_starting_at_level_two_is_numbered_from_one(): def test_a_main_document_opening_below_level_one_skips_a_level(body, level): """§4.1: a document with frontmatter is a main document, whose first heading is at level 1.""" - result = validate_document(parse_document(_BARE + body)) + result = validate(parse(_BARE + body)) assert [d.message for d in result.diagnostics if d.rule == "heading-skip"] == [ f"Heading levels must not skip. 'A' is at level {level}, but the document has no level-1 heading before it." ] @@ -725,7 +724,7 @@ def test_a_main_document_opening_below_level_one_skips_a_level(body, level): ], ) def test_where_a_first_heading_below_level_one_is_a_skip(source, skips): - assert ("heading-skip" in validate_document(parse_document(source)).rules()) is skips + assert ("heading-skip" in validate(parse(source)).rules()) is skips def test_numbers_are_unique_and_unchanged_without_a_skip(): @@ -749,18 +748,18 @@ def test_numbers_are_unique_and_unchanged_without_a_skip(): def test_out_of_range_heading_keeps_its_index_entry(): - """result.sections stays positionally paired with document.sections. + """result.index.sections stays positionally paired with document.sections. Consumers zip the two; dropping an entry for an invalid heading level shifts every later section's number and loses the last one entirely. """ body = "# One {#one}\n\nText.\n\n###### Deep {#deep}\n\nText.\n\n# Two {#two}\n\nText.\n" source = _FRONTMATTER.replace("# Terms {#terms}\n\n", "") + body - document = parse_document(source, filename="t.lgd") - result = validate_document(document) + document = parse(source, filename="t.lgd") + result = validate(document) assert "heading-depth" in result.rules("error") - assert len(result.sections) == len(document.sections) - assert [e.title for e in result.sections] == [s.title for s in document.sections] + assert len(result.index.sections) == len(document.sections) + assert [e.title for e in result.index.sections] == [s.title for s in document.sections] def test_amend_term_is_an_error_when_original_defines_nothing(): @@ -769,8 +768,8 @@ def test_amend_term_is_an_error_when_original_defines_nothing(): "document_type: contract", "document_type: contract\namends:\n title: Original\n file: original.lgd", ) + "Uses {{term: missing}}.\n" - document = parse_document(source, filename="amendment.lgd") - result = validate_document(document, import_definitions=lambda *_: {}) + document = parse(source, filename="amendment.lgd") + result = validate(document, resolve=lambda path: "---\ntitle: Original\n---\n") assert "amend-term-undefined" in result.rules("error") assert "amend-term-unresolvable" not in result.rules() @@ -784,10 +783,10 @@ def test_inline_value_indices_collect_field_spec_values(): "Rate {{field: 1.5%, type=percentage}} over {{duration: 30, unit=MIN}} " "from {{date: 2026-06-01}} for {{money: 5000, currency=EUR}}." ) - assert ("1.5%", "percentage") in result.inline_fields - assert ("30", "MIN") in result.inline_durations - assert "2026-06-01" in result.inline_dates - assert ("5000", "EUR") in result.inline_money + assert ("1.5%", "percentage") in result.index.values.fields + assert ("30", "MIN") in result.index.values.durations + assert "2026-06-01" in result.index.values.dates + assert ("5000", "EUR") in result.index.values.money # ── Attachments (§16.10) ────────────────────────────────────────── @@ -819,7 +818,7 @@ def test_inline_value_indices_collect_field_spec_values(): def _validate_attachments(attachments: str, body: str = "Text."): source = _ATTACHMENT_FRONTMATTER.format(attachments=attachments, body=body) - return validate_document(parse_document(source, filename="t.lgd")) + return validate(parse(source, filename="t.lgd")) def test_duplicate_attachment_id_is_reported(): @@ -885,9 +884,10 @@ def test_amendment_redefining_an_imported_term_is_a_warning(): The {{term: services}} are amended. """ - document = parse_document(source, filename="first-amendment.lgd") - result = validate_document( - document, import_definitions=lambda *_: {"services": "Services"} + document = parse(source, filename="first-amendment.lgd") + result = validate( + document, + resolve=lambda path: '---\ntitle: Original\n---\n\n# A\n\n"Services" {{def: services}} means x.\n', ) assert "amend-def-override" in result.rules("warning") @@ -905,34 +905,34 @@ def test_amendment_redefining_an_imported_term_is_a_warning(): def test_preamble_is_kept_out_of_the_numbered_sections(): - document = parse_document(_PREAMBLE_SOURCE) + document = parse(_PREAMBLE_SOURCE) assert [s.identifier for s in document.sections] == ["confidentiality"] assert "entered into\nbetween" in document.preamble[0].text def test_definition_in_the_preamble_is_document_wide(): - result = validate_document(parse_document(_PREAMBLE_SOURCE)) - assert result.definition_lookup == {"agreement": "Agreement"} + result = validate(parse(_PREAMBLE_SOURCE)) + assert result.index.definition_lookup == {"agreement": "Agreement"} assert result.diagnostics == [] def test_directives_in_the_preamble_are_checked(): source = _PREAMBLE_SOURCE.replace("{{party: beta}}", "{{party: nobody}} on {{date: 2026-02-30}}") - assert {"party-unknown", "date-invalid"} <= validate_document(parse_document(source)).rules("error") + assert {"party-unknown", "date-invalid"} <= validate(parse(source)).rules("error") def test_preamble_round_trips(): - document = parse_document(_PREAMBLE_SOURCE) - reparsed = parse_document(serialize_document(document)) + document = parse(_PREAMBLE_SOURCE) + reparsed = parse(serialize_document(document)) assert reparsed.preamble == document.preamble assert reparsed.sections == document.sections def test_document_of_only_a_preamble(): source = _FRONTMATTER.replace("# Terms {#terms}\n\n", "") + "By {{party: nobody}}.\n" - document = parse_document(source) + document = parse(source) assert document.sections == [] and len(document.preamble) == 1 - assert "party-unknown" in validate_document(document).rules("error") + assert "party-unknown" in validate(document).rules("error") @pytest.mark.parametrize("marker", ["{#intro}", "{#Bad_ID}"]) @@ -941,14 +941,14 @@ def test_preamble_paragraph_anchor_is_misplaced(marker): source = _PREAMBLE_SOURCE.replace("{{party: beta}}.", "{{party: beta}}. " + marker) + ( "See {{ref: intro}}.\n" ) - result = validate_document(parse_document(source)) + result = validate(parse(source)) assert "anchor-misplaced" in result.rules("warning") assert "ref-broken" in result.rules("error") assert "anchor-format" not in result.rules() def test_preamble_survives_the_dict_round_trip(): - document = parse_document(_PREAMBLE_SOURCE) + document = parse(_PREAMBLE_SOURCE) assert document_from_dict(document_to_dict(document)) == document ref = next(r for r in collect_definitions(document) if r.id == "agreement") assert ref.section_index is None @@ -1028,7 +1028,7 @@ def test_paragraph_anchor_colliding_with_an_attachment_id_is_reported(): def test_empty_frontmatter_and_byte_order_mark_are_not_body(source): """Empty frontmatter stops at its own closing ---, not at a later rule, and a byte-order mark never hides the first heading.""" - document = parse_document(source) + document = parse(source) assert document.preamble == [] assert document.sections[0].identifier == "a" @@ -1039,19 +1039,19 @@ def test_empty_frontmatter_and_byte_order_mark_are_not_body(source): def _presence(source: str) -> tuple[bool, list[str], list[str], set[str]]: """Whether the frontmatter is absent, the preamble's block kinds, the section titles, and the rules reported.""" - document = parse_document(source) + document = parse(source) return ( document.metadata.frontmatter_absent, [b.kind for b in document.preamble], [s.title for s in document.sections], - validate_document(document).rules(), + validate(document).rules(), ) def test_a_document_without_frontmatter_draws_only_the_warning(): """§16.6: not title-missing, not sides-absent.""" assert _presence("# Scope {#scope}\n\nBody.\n") == (True, [], ["Scope"], {"frontmatter-absent"}) - result = validate_document(parse_document("# Scope\n\nBody.\n")) + result = validate(parse("# Scope\n\nBody.\n")) assert [d.level for d in result.diagnostics] == ["warning"] @@ -1083,12 +1083,12 @@ def test_empty_frontmatter_is_present(block): def test_invalid_yaml_is_still_an_error(): with pytest.raises(yaml.YAMLError): - parse_document("---\ntitle: [unclosed\n---\n# A\n") + parse("---\ntitle: [unclosed\n---\n# A\n") def test_a_mapping_of_another_type_is_not_frontmatter_fields(): with pytest.raises(ValueError, match="mapping of fields"): - parse_document("---\n!!set {title: T}\n---\n# A\n") + parse("---\n!!set {title: T}\n---\n# A\n") @pytest.mark.parametrize("frontmatter", ["!!set {title: T}", "title: [x", "a: " + "{" * 3000]) @@ -1098,7 +1098,7 @@ def test_frontmatter_that_cannot_be_read_raises_frontmatter_error(frontmatter): from legaldown import FrontmatterError with pytest.raises(FrontmatterError) as raised: - parse_document(f"---\n{frontmatter}\n---\n# A\n") + parse(f"---\n{frontmatter}\n---\n# A\n") assert isinstance(raised.value, yaml.YAMLError) and isinstance(raised.value, ValueError) @@ -1112,24 +1112,24 @@ def test_frontmatter_that_cannot_be_read_raises_frontmatter_error(frontmatter): ], ) def test_a_document_without_frontmatter_is_written_without_it(source): - document = parse_document(source) + document = parse(source) written = serialize_document(document) assert not written.startswith("---") - assert parse_document(written) == document - assert validate_document(parse_document(written)).rules() <= {"frontmatter-absent"} + assert parse(written) == document + assert validate(parse(written)).rules() <= {"frontmatter-absent"} def test_metadata_set_on_a_bare_document_is_written_as_frontmatter(): - document = parse_document("# A\n\nText.\n") + document = parse("# A\n\nText.\n") document.metadata.title = "Terms" assert serialize_document(document).startswith("---\ntitle: Terms\n") def test_frontmatter_absent_is_not_read_from_frontmatter_or_a_dict(): - document = parse_document("---\ntitle: T\nfrontmatter_absent: true\n---\n\n# A\n") + document = parse("---\ntitle: T\nfrontmatter_absent: true\n---\n\n# A\n") assert document.metadata.frontmatter_absent is False - assert document_from_dict(document_to_dict(parse_document("# A\n"))).metadata.frontmatter_absent is False - assert "frontmatter-absent" not in validate_document(document_from_dict({"sections": []})).rules() + assert document_from_dict(document_to_dict(parse("# A\n"))).metadata.frontmatter_absent is False + assert "frontmatter-absent" not in validate(document_from_dict({"sections": []})).rules() def test_only_a_comment_may_follow_an_anchor(): @@ -1140,7 +1140,7 @@ def test_only_a_comment_may_follow_an_anchor(): "Per {{ref: delivery}}." ) assert [d.rule for d in result.diagnostics] == ["anchor-misplaced"] - assert "delivery" in result.section_lookup + assert "delivery" in result.index.section_lookup def test_marker_after_a_malformed_directive_is_still_an_anchor(): @@ -1156,7 +1156,7 @@ def test_marker_after_a_malformed_directive_is_still_an_anchor(): def _outline(source: str) -> list[tuple[str, int, str]]: """Each heading's text, level, and identifier: explicit or generated.""" - entries = validate_document(parse_document(source)).sections + entries = validate(parse(source)).index.sections return [(entry.title, entry.level, entry.identifier) for entry in entries] @@ -1166,18 +1166,18 @@ def _outline(source: str) -> list[tuple[str, int, str]]: def test_only_spaces_and_tabs_and_nine_digits_make_a_heading_or_an_item(line): """CommonMark: other whitespace and longer or non-ASCII numbers are paragraph text (cmark-gfm).""" - document = parse_document(_FRONTMATTER + f"{line}\n") + document = parse(_FRONTMATTER + f"{line}\n") assert [(b.kind, b.text) for b in document.sections[0].blocks] == [ ("paragraph", line.strip(" \t\n\r")) ] - assert parse_document(serialize_document(document)) == document + assert parse(serialize_document(document)) == document @pytest.mark.parametrize(("line", "kind"), [ ("#\tTitle", None), ("123456789. item", "ordered_list"), ("-\titem", "unordered_list"), ]) def test_a_tab_or_nine_digits_still_make_a_heading_or_an_item(line, kind): - document = parse_document(_FRONTMATTER + f"{line}\n") + document = parse(_FRONTMATTER + f"{line}\n") if kind is None: assert [s.title for s in document.sections] == ["Terms", "Title"] else: @@ -1188,7 +1188,7 @@ def test_setext_headings_are_headings(): """§4.1: === and --- underlines make level-1 and level-2 headings.""" source = _BARE + "Intro.\n\nPayment\nTerms {#pay}\n=======\n\nText.\n\nLate Fees\n---\n\nMore.\n" assert _outline(source) == [("Payment Terms", 1, "pay"), ("Late Fees", 2, "late-fees")] - assert len(parse_document(source).preamble) == 1 + assert len(parse(source).preamble) == 1 @pytest.mark.parametrize("character", ["\x0c", "\x0b", "\x1c", "\x85", "\u2028"]) @@ -1200,7 +1200,7 @@ def test_only_lf_cr_and_crlf_end_a_line(character): def test_an_unclosed_fence_at_the_end_holds_no_extra_line(): - document = parse_document(_FRONTMATTER + "```\ncode\n") + document = parse(_FRONTMATTER + "```\ncode\n") assert document.sections[0].blocks[0].text == "```\ncode" @@ -1234,14 +1234,14 @@ def test_a_list_item_or_quote_that_is_one_fence_line_is_code(body): def test_a_nested_item_interrupts_its_items_paragraph_only_as_commonmark_allows(body, items): """Only a bullet with text, or an ordered item numbered 1, may interrupt a paragraph — the item's own too (cmark-gfm).""" - document = parse_document(_FRONTMATTER + body) + document = parse(_FRONTMATTER + body) assert [(b.kind, _texts(b)) for b in document.sections[0].blocks] == items @pytest.mark.parametrize("number", ["1", "01", "001"]) def test_an_item_numbered_1_however_written_interrupts_a_paragraph(number): """#57: the number is compared as a number.""" - document = parse_document(_FRONTMATTER + f"Text\n{number}. item\n") + document = parse(_FRONTMATTER + f"Text\n{number}. item\n") assert [b.kind for b in document.sections[0].blocks] == ["paragraph", "ordered_list"] @@ -1255,8 +1255,8 @@ def test_an_anchor_after_a_heading_in_an_item_is_misplaced(): @pytest.mark.parametrize("text", [" See {{ref: x}}.", "See {{ref: x}} ", " Use {{term: x}}."]) def test_a_no_break_space_round_trips_around_a_lifted_directive(text): - document = parse_document(_FRONTMATTER + text + "\n") - assert parse_document(serialize_document(document)) == document + document = parse(_FRONTMATTER + text + "\n") + assert parse(serialize_document(document)) == document def test_a_paragraph_of_only_spaces_is_empty(): @@ -1268,7 +1268,7 @@ def test_a_paragraph_of_a_no_break_space_is_text(): # Not blank in CommonMark (#47): written back, it reads the same. document = document_from_dict({"sections": [{"title": "A", "blocks": [{"kind": "paragraph", "text": "\u00a0"}]}]}) assert document.sections[0].blocks[0].text == "\u00a0" - written = parse_document(serialize_document(document)) + written = parse(serialize_document(document)) assert [(b.kind, b.text) for b in written.sections[0].blocks] == [("paragraph", "\u00a0")] @@ -1296,11 +1296,11 @@ def test_a_marker_on_an_items_later_paragraph_is_misplaced(): ("- a\n\n b\n", ["unordered_list", "paragraph"], ["Terms"]), ]) def test_where_a_list_does_not_continue_past_a_blank_line(body, kinds, titles): - document = parse_document(_FRONTMATTER + body) + document = parse(_FRONTMATTER + body) assert [b.kind for b in document.sections[0].blocks] == kinds assert [s.title for s in document.sections] == titles if "- \n" not in body: # an empty item is not written back (#46) - assert parse_document(serialize_document(document)) == document + assert parse(serialize_document(document)) == document @pytest.mark.parametrize(("body", "items", "after"), [ @@ -1312,7 +1312,7 @@ def test_where_a_list_does_not_continue_past_a_blank_line(body, kinds, titles): ("- a\n\n 2. b\n", ["a\n1. b"], []), ]) def test_what_an_items_later_content_is(body, items, after): - document = parse_document(_FRONTMATTER + body) + document = parse(_FRONTMATTER + body) blocks = document.sections[0].blocks assert _texts(blocks[0]) == items and [b.kind for b in blocks[1:]] == after @@ -1323,10 +1323,10 @@ def test_what_an_items_later_content_is(body, items, after): ]) def test_a_line_short_of_every_open_items_content_is_code(body): """No item reaches it, so it is indented code (cmark-gfm).""" - document = parse_document(_FRONTMATTER + body) + document = parse(_FRONTMATTER + body) assert [b.kind for b in document.sections[0].blocks] == ["ordered_list" if body[0] == "1" else "unordered_list", "code"] assert "ref-broken" not in _validate(body).rules() - assert parse_document(serialize_document(document)) == document + assert parse(serialize_document(document)) == document @pytest.mark.parametrize("body", [ @@ -1337,14 +1337,14 @@ def test_a_line_short_of_every_open_items_content_is_code(body): def test_a_block_after_a_wide_list_marker_round_trips(body): """The list is written back with its last item's content past the block's indentation, so the block stays after it.""" - document = parse_document(_FRONTMATTER + body) - assert parse_document(serialize_document(document)) == document + document = parse(_FRONTMATTER + body) + assert parse(serialize_document(document)) == document @pytest.mark.parametrize("text", ["
", "", ""]) def test_a_tag_is_split_only_where_both_lines_stay_paragraph_text(text): document = document_from_dict({"sections": [{"title": "A", "blocks": [{"kind": "paragraph", "text": text}]}]}) - blocks = parse_document(serialize_document(document)).sections[0].blocks + blocks = parse(serialize_document(document)).sections[0].blocks assert [b.kind for b in blocks] == ["paragraph"] @@ -1360,15 +1360,15 @@ def test_a_tag_is_split_only_where_both_lines_stay_paragraph_text(text): ("- a\n- \n\n code\n", ["unordered_list", "code"], ["Terms"]), ]) def test_what_follows_a_list_item_that_holds_no_open_paragraph(body, kinds, titles): - document = parse_document(_FRONTMATTER + body) + document = parse(_FRONTMATTER + body) assert [b.kind for b in document.sections[0].blocks] == kinds assert [s.title for s in document.sections] == titles - assert parse_document(serialize_document(document)) == document + assert parse(serialize_document(document)) == document def test_wide_marker_spacing_sets_the_items_column_as_commonmark_does(): """Five spaces after the marker: the content column is one past it.""" - document = parse_document(_FRONTMATTER + "- foo\n\n bar\n") + document = parse(_FRONTMATTER + "- foo\n\n bar\n") # The rest of the line is indented code, and so is `bar`: " foo" and # "bar" (cmark-gfm). [block] = document.sections[0].blocks @@ -1380,12 +1380,12 @@ def test_an_unclosed_fence_after_a_list_is_closed_when_written(): {"title": "A", "blocks": [{"kind": "unordered_list", "items": ["a"]}, {"kind": "code", "text": "```\nx"}]}, {"title": "B", "blocks": []}, ]}) - assert [s.title for s in parse_document(serialize_document(document)).sections] == ["A", "B"] + assert [s.title for s in parse(serialize_document(document)).sections] == ["A", "B"] def test_an_empty_list_does_not_take_the_code_after_it(): - document = parse_document(_FRONTMATTER + "- a\n\n* \n\n code\n") - reparsed = parse_document(serialize_document(document)) + document = parse(_FRONTMATTER + "- a\n\n* \n\n code\n") + reparsed = parse(serialize_document(document)) assert [(b.kind, _texts(b), b.text) for b in reparsed.sections[0].blocks] == [ ("unordered_list", ["a"], ""), ("unordered_list", [""], ""), ("code", [], " code") ] @@ -1393,7 +1393,7 @@ def test_an_empty_list_does_not_take_the_code_after_it(): def _placeholders(body: str) -> int: - document = parse_document(_FRONTMATTER + body) + document = parse(_FRONTMATTER + body) return sum( 1 for _s, _i, block in document.iter_blocks() for fragment in text_fragments(block) for directive in lex(fragment).directives if directive.name == "placeholder" and not directive.malformed @@ -1425,8 +1425,8 @@ def test_an_items_later_content_is_read_as_commonmark_reads_it(body, count): def test_an_items_later_content_round_trips(body): """A `#` line in it is written in far enough not to read as a heading; a fence left open in it ends with the item and is not closed.""" - document = parse_document(_FRONTMATTER + body) - assert parse_document(serialize_document(document)) == document + document = parse(_FRONTMATTER + body) + assert parse(serialize_document(document)) == document def test_a_table_cell_opening_like_a_fence_is_text(): @@ -1434,13 +1434,13 @@ def test_a_table_cell_opening_like_a_fence_is_text(): def test_frontmatter_with_cr_line_endings_is_frontmatter(): - document = parse_document("---\rtitle: X\r---\r\r# A\r") + document = parse("---\rtitle: X\r---\r\r# A\r") assert not document.metadata.frontmatter_absent assert document.metadata.title == "X" def test_a_quote_round_trips_a_character_that_is_not_a_line_ending(): - document = parse_document(_FRONTMATTER + "> One\x0ctwo.\n> Three.\n") + document = parse(_FRONTMATTER + "> One\x0ctwo.\n> Three.\n") assert serialize_document(document).endswith("> One\x0ctwo.\n> Three.\n") @@ -1449,7 +1449,7 @@ def test_a_quote_round_trips_a_character_that_is_not_a_line_ending(): ["Text.\n\n---\n\nMore.\n", "- item\n---\n", "> quoted\n---\n", "| a | b |\n|---|---|\n| c | d |\n---\n"], ) def test_dashes_not_under_a_paragraph_are_a_rule(body): - document = parse_document(_FRONTMATTER + body) + document = parse(_FRONTMATTER + body) assert _outline(_FRONTMATTER + body) == [("Terms", 1, "terms")] assert "rule" in [b.kind for b in document.sections[0].blocks] @@ -1462,26 +1462,26 @@ def test_a_signature_block_heading_is_an_ordinary_section(hashes): f"{_BARE}# A\n\nSee {{{{ref: signature-block}}}}.\n\n" f"{hashes} Signature Block {{#signature-block}}\n\nSigned by the parties. {{{{ref: nowhere}}}}\n\n# After\n\nText.\n" ) - document = parse_document(source) + document = parse(source) assert [s.title for s in document.sections] == ["A", "Signature Block", "After"] - result = validate_document(document) + result = validate(document) assert [d.message for d in result.diagnostics if d.rule == "ref-broken"] == [ "Broken section reference: 'nowhere'." ] serialized = serialize_document(document) assert "Signed by the parties." in serialized - assert parse_document(serialized) == document + assert parse(serialized) == document def test_a_second_signature_block_identifier_is_a_duplicate(): source = _BARE + "# Signature Block {#signature-block}\n\nA.\n\n# Signature Block {#signature-block}\n\nB.\n" - assert "anchor-duplicate" in validate_document(parse_document(source)).rules() + assert "anchor-duplicate" in validate(parse(source)).rules() def test_setext_heading_round_trips_as_an_atx_heading(): """Its generated identifier is not written: it stays generated.""" source = _BARE + "Scope\n=====\n\nText.\n" - assert "\n# Scope\n" in serialize_document(parse_document(source)) + assert "\n# Scope\n" in serialize_document(parse(source)) def test_fenced_code_is_one_literal_block(): @@ -1489,10 +1489,10 @@ def test_fenced_code_is_one_literal_block(): blank line inside it does not end it.""" fence = "```\n# not a heading\n\n{#z} {{ref: nope}} {{\n```" source = _FRONTMATTER + fence + "\n\n# Next {#next}\n" - document = parse_document(source) + document = parse(source) assert _outline(source) == [("Terms", 1, "terms"), ("Next", 1, "next")] assert [(b.kind, b.text) for b in document.sections[0].blocks] == [("code", fence)] - assert validate_document(document).diagnostics == [] + assert validate(document).diagnostics == [] assert fence in serialize_document(document) @@ -1516,13 +1516,13 @@ def test_inline_triple_backticks_do_not_open_a_fence(): def test_fence_interrupts_a_paragraph(): - document = parse_document(_FRONTMATTER + "Example:\n```\n{{ref: nope}}\n```\n") + document = parse(_FRONTMATTER + "Example:\n```\n{{ref: nope}}\n```\n") assert [b.kind for b in document.sections[0].blocks] == ["paragraph", "code"] - assert validate_document(document).diagnostics == [] + assert validate(document).diagnostics == [] def _kinds(source: str) -> list[str]: - return [b.kind for b in parse_document(source).sections[0].blocks] + return [b.kind for b in parse(source).sections[0].blocks] @pytest.mark.parametrize( @@ -1549,9 +1549,9 @@ def test_dashes_after_a_lazy_continuation_are_a_rule(body): def test_fence_inside_a_list_item_does_not_swallow_the_section(): body = "- Example:\n ```\n code\n\n code2\n ```\n\nSee {{ref: nope}}.\n" - document = parse_document(_FRONTMATTER + body) + document = parse(_FRONTMATTER + body) assert [b.kind for b in document.sections[0].blocks] == ["unordered_list", "ref"] - assert "ref-broken" in validate_document(document).rules("error") + assert "ref-broken" in validate(document).rules("error") def test_tab_indented_backticks_neither_open_nor_close_a_fence(): @@ -1563,7 +1563,7 @@ def test_tab_indented_backticks_neither_open_nor_close_a_fence(): def test_indented_fence_round_trips_unchanged(): fence = " ```\n code\n ```" - assert fence in serialize_document(parse_document(_FRONTMATTER + "Para.\n\n" + fence + "\n")) + assert fence in serialize_document(parse(_FRONTMATTER + "Para.\n\n" + fence + "\n")) def test_paragraph_directly_above_dashes_is_a_setext_heading(): @@ -1576,16 +1576,16 @@ def test_paragraph_directly_above_dashes_is_a_setext_heading(): def test_fence_opening_on_a_list_marker_line_stays_in_the_item(): body = "- ```\n a\n\n b\n ```\n\n# Next {#next}\n\nSee {{ref: nope}}.\n" - document = parse_document(_FRONTMATTER + body) + document = parse(_FRONTMATTER + body) assert _outline(_FRONTMATTER + body) == [("Terms", 1, "terms"), ("Next", 1, "next")] assert _texts(document.sections[0].blocks[0]) == ["```\na\n\nb\n```"] - assert "ref-broken" in validate_document(document).rules("error") + assert "ref-broken" in validate(document).rules("error") def test_fence_in_a_list_item_is_literal_and_round_trips(): body = "- Example:\n ~~~\n {{ref: nope}}\n\n {#zz}\n ~~~\n" - document = parse_document(_FRONTMATTER + body) - assert validate_document(document).diagnostics == [] + document = parse(_FRONTMATTER + body) + assert validate(document).diagnostics == [] assert body.strip() in serialize_document(document) @@ -1601,9 +1601,9 @@ def test_indented_dashes_are_code(): def test_a_single_pipe_line_is_a_paragraph_not_dropped(): - document = parse_document(_FRONTMATTER + "| lone {{ref: nope}}\n") + document = parse(_FRONTMATTER + "| lone {{ref: nope}}\n") assert _kinds(_FRONTMATTER + "| lone {{ref: nope}}\n") == ["ref"] - assert "ref-broken" in validate_document(document).rules("error") + assert "ref-broken" in validate(document).rules("error") def test_fence_in_a_block_quote_is_literal(): @@ -1612,9 +1612,9 @@ def test_fence_in_a_block_quote_is_literal(): def test_code_block_without_a_fence_is_checked_as_text(): """Only a fenced block is literal; a hand-built one is not taken on trust.""" - document = parse_document(_FRONTMATTER + "Text.\n") + document = parse(_FRONTMATTER + "Text.\n") document.sections[0].blocks.append(Block(kind="code", text="See {{ref: nope}}.")) - assert "ref-broken" in validate_document(document).rules("error") + assert "ref-broken" in validate(document).rules("error") @pytest.mark.parametrize( @@ -1634,21 +1634,21 @@ def test_list_or_quote_interrupts_a_paragraph(body, kinds): def test_item_text_after_its_closed_fence_is_checked(): - document = parse_document(_FRONTMATTER + "- ```\n code\n ```\n more {{ref: nowhere}}\n") + document = parse(_FRONTMATTER + "- ```\n code\n ```\n more {{ref: nowhere}}\n") assert _texts(document.sections[0].blocks[0]) == ["```\ncode\n```\nmore {{ref: nowhere}}"] - assert "ref-broken" in validate_document(document).rules("error") - assert parse_document(serialize_document(document)).sections == document.sections + assert "ref-broken" in validate(document).rules("error") + assert parse(serialize_document(document)).sections == document.sections def test_unclosed_fence_in_an_item_ends_at_the_next_item(): - document = parse_document(_FRONTMATTER + "1. ```\n x\n2. b\n3. c\n") + document = parse(_FRONTMATTER + "1. ```\n x\n2. b\n3. c\n") assert [(b.kind, _texts(b)) for b in document.sections[0].blocks] == [ ("ordered_list", ["```\nx", "b", "c"]) ] def test_tab_after_a_list_marker_counts_as_columns(): - document = parse_document(_FRONTMATTER + "-\t```\n\tx\n\t```\n") + document = parse(_FRONTMATTER + "-\t```\n\tx\n\t```\n") assert _texts(document.sections[0].blocks[0]) == ["```\nx\n```"] @@ -1665,23 +1665,23 @@ def test_rescan_after_a_directive_starts_at_its_line(): def test_code_block_text_after_its_closing_fence_is_checked(): - document = parse_document(_FRONTMATTER + "Text.\n") + document = parse(_FRONTMATTER + "Text.\n") document.sections[0].blocks.append(Block(kind="code", text="```\nx\n```\n{{ref: missing}}")) - assert "ref-broken" in validate_document(document).rules("error") + assert "ref-broken" in validate(document).rules("error") def test_serializer_closes_an_unclosed_fence(): - document = parse_document(_FRONTMATTER + "Text.\n\n# Next {#next}\n") + document = parse(_FRONTMATTER + "Text.\n\n# Next {#next}\n") document.sections[0].blocks.append(Block(kind="code", text="```\nx")) - reparsed = parse_document(serialize_document(document)) + reparsed = parse(serialize_document(document)) assert [s.identifier for s in reparsed.sections] == ["terms", "next"] def test_code_in_a_list_item_keeps_trailing_spaces(): source = _FRONTMATTER + "- ```\n keep \n ```\n" - item = parse_document(source).sections[0].blocks[0].items[0] + item = parse(source).sections[0].blocks[0].items[0] assert [(block.kind, block.text) for block in item.blocks] == [("code", "```\nkeep \n```")] - assert parse_document(serialize_document(parse_document(source))).sections[0].blocks[0].items[0] == item + assert parse(serialize_document(parse(source))).sections[0].blocks[0].items[0] == item @pytest.mark.parametrize( @@ -1699,32 +1699,32 @@ def test_backticks_in_directive_values_open_nothing(text, targets): @pytest.mark.parametrize("paragraph", [" ~~~\n x", " # not a heading"]) def test_paragraph_that_would_open_a_block_stays_indented(paragraph): source = _FRONTMATTER + paragraph + "\n\n# Next {#next}\n\nBody.\n" - reparsed = parse_document(serialize_document(parse_document(source))) - assert reparsed.sections == parse_document(source).sections + reparsed = parse(serialize_document(parse(source))) + assert reparsed.sections == parse(source).sections def test_tabs_inside_list_item_code_are_kept(): - document = parse_document(_FRONTMATTER + "- item\n ```\n a\tb\n ```\n") + document = parse(_FRONTMATTER + "- item\n ```\n a\tb\n ```\n") assert _texts(document.sections[0].blocks[0]) == ["item\n```\na\tb\n```"] def test_a_fence_left_open_in_a_list_item_ends_with_its_list(): # The next list takes another bullet: its marker ends the fence and the # list, so the fence is written as it is. - document = parse_document(_FRONTMATTER + "Text.\n") + document = parse(_FRONTMATTER + "Text.\n") document.sections[0].blocks += [ Block(kind="unordered_list", items=["x\n```\ncode"]), Block(kind="unordered_list", items=["y"]), ] - blocks = parse_document(serialize_document(document)).sections[0].blocks + blocks = parse(serialize_document(document)).sections[0].blocks assert [_texts(b) for b in blocks[1:]] == [["x\n```\ncode"], ["y"]] def test_indentation_after_the_quote_marker_is_content(): """Only one space after > is syntax; a fence line indented four columns inside quoted code is content, not a closing fence.""" - document = parse_document(_FRONTMATTER + "> ```\n> ```\n> {{ref: missing}}\n> ```\n") - assert validate_document(document).diagnostics == [] + document = parse(_FRONTMATTER + "> ```\n> ```\n> {{ref: missing}}\n> ```\n") + assert validate(document).diagnostics == [] @pytest.mark.parametrize( @@ -1745,8 +1745,8 @@ def test_only_some_list_items_interrupt_a_paragraph(second_line, kinds, headings def _blocks(body: str) -> list[tuple[str, str | list[str]]]: - document = parse_document(_FRONTMATTER + body) - assert parse_document(serialize_document(document)).sections == document.sections + document = parse(_FRONTMATTER + body) + assert parse(serialize_document(document)).sections == document.sections return [(b.kind, _texts(b) if b.kind.endswith("list") else b.text) for b in document.sections[0].blocks] @@ -1776,7 +1776,7 @@ def test_an_item_beginning_with_dashes_is_not_written_as_a_rule(body): def test_a_nested_item_of_another_type_stays_in_the_list(): - [block] = parse_document(_FRONTMATTER + "- parent\n * child\n 1) child\n- next\n").sections[0].blocks + [block] = parse(_FRONTMATTER + "- parent\n * child\n 1) child\n- next\n").sections[0].blocks assert _shape(block) == ("unordered_list", [ ["parent", ("unordered_list", [["child"]]), ("ordered_list", [["child"]])], ["next"] ]) @@ -1822,13 +1822,13 @@ def test_a_drafting_note_in_a_star_item_is_recognized(): def _table(body: str) -> Block: - [block] = parse_document(_FRONTMATTER + body).sections[0].blocks + [block] = parse(_FRONTMATTER + body).sections[0].blocks return block def _round_trips(body: str) -> None: - document = parse_document(_FRONTMATTER + body) - assert parse_document(serialize_document(document)).sections == document.sections + document = parse(_FRONTMATTER + body) + assert parse(serialize_document(document)).sections == document.sections def test_an_escaped_pipe_is_cell_text_and_alignment_is_kept(): @@ -1838,7 +1838,7 @@ def test_an_escaped_pipe_is_cell_text_and_alignment_is_kept(): assert block.align == ["left", "right", "center", ""] assert block.rows == [["1", "2", "3", "4"]] assert "| a \\| b | c | d | e |\n| :--- | ---: | :---: | --- |" in serialize_document( - parse_document(_FRONTMATTER + body) + parse(_FRONTMATTER + body) ) _round_trips(body) @@ -1899,7 +1899,7 @@ def test_table_cells_are_positional_in_the_model(): ]}]} ).sections[0].blocks[0] assert (block.headers, block.rows, block.align) == (["", "b"], [["", ""], ["1", ""], ["1", "2"]], ["right", ""]) - assert document_from_dict(document_to_dict(parse_document( + assert document_from_dict(document_to_dict(parse( _FRONTMATTER + "| a | b |\n|:--|--:|\n" ))).sections[0].blocks[0].align == ["left", "right"] assert document_from_dict({"sections": [{"blocks": [{"kind": "table"}]}]}).sections[0].blocks[0].rows == [["", ""]] @@ -1921,10 +1921,10 @@ def written(block: Block) -> list[str]: assert written({"kind": "table", "headers": [], "rows": [[]]}) == ["| |", "| --- |", "| |"] block = {"kind": "table", "headers": ["a\\|b", "c\\\\|"], "rows": [["\\", "|"]]} document = document_from_dict({"sections": [{"title": "A", "blocks": [block]}]}) - assert parse_document(serialize_document(document)).sections == document.sections + assert parse(serialize_document(document)).sections == document.sections # Nothing to take a width from: one empty column, still a table. document = document_from_dict({"sections": [{"title": "A", "blocks": [{"kind": "table", "headers": [], "rows": [[]]}]}]}) - assert parse_document(serialize_document(document)).sections[0].blocks[0].kind == "table" + assert parse(serialize_document(document)).sections[0].blocks[0].kind == "table" def test_default_headers_widen_to_the_widest_row(): @@ -1939,7 +1939,7 @@ def test_default_headers_widen_to_the_widest_row(): ) def test_a_table_interrupts_a_paragraph(body, kinds): """GFM: a paragraph's last line and a delimiter row under it start a table.""" - document = parse_document(_FRONTMATTER + body) + document = parse(_FRONTMATTER + body) assert [b.kind for b in document.sections[0].blocks] == kinds _round_trips(body) @@ -1950,8 +1950,8 @@ def test_a_table_interrupts_a_paragraph(body, kinds): def _html(body: str, *, bare: bool = False) -> tuple[list[str], list[tuple[str, str]]]: """Section titles, and the (kind, text) of every block, preamble first.""" source = (_BARE if bare else _FRONTMATTER) + body - document = parse_document(source) - assert parse_document(serialize_document(document)) == document + document = parse(source) + assert parse(serialize_document(document)) == document blocks = [(b.kind, b.text) for _section, _index, b in document.iter_blocks()] return [s.title for s in document.sections], blocks @@ -1994,14 +1994,14 @@ def test_a_whole_line_after_a_comment_is_raw_html(): """CommonMark: the line that ends a comment block belongs to it, so its text is not rendered and its directives are not recognized (§11.4).""" source = _FRONTMATTER + " The Buyer pays {{money: 5}} under {{ref: nope}}.\n" - result = validate_document(parse_document(source)) + result = validate(parse(source)) # Raw HTML, not a comment: rendered nowhere, and warned about (§8.7). - assert result.rules() == {"raw-html"} and result.inline_money == [] + assert result.rules() == {"raw-html"} and result.index.values.money == [] def test_nothing_in_an_html_block_is_validated(): source = _FRONTMATTER + '
{{ref: nope}} {{bogus: x}} {{ "Fee" {{def: fee}}\n{#anchor}
\n\nSee {{ref: anchor}}.\n' - assert validate_document(parse_document(source)).rules() == {"ref-broken", "raw-html"} + assert validate(parse(source)).rules() == {"ref-broken", "raw-html"} @pytest.mark.parametrize( @@ -2037,7 +2037,7 @@ def test_an_html_block_in_the_model_keeps_its_indentation(): {"sections": [{"title": "A", "blocks": [{"kind": "html", "text": "\n\n
\n x\n
\n\n"}]}]} ) assert document.sections[0].blocks[0].text == "
\n x\n
" - assert parse_document(serialize_document(document)).sections == document.sections + assert parse(serialize_document(document)).sections == document.sections @pytest.mark.parametrize("prefix", ["
see", "# see", "```"]) @@ -2050,7 +2050,7 @@ def test_a_model_built_reference_that_would_open_a_block_is_escaped(prefix): {"kind": "paragraph", "text": f"{prefix} text"}, ]}]} ) - reparsed = parse_document(serialize_document(document)).sections[0].blocks + reparsed = parse(serialize_document(document)).sections[0].blocks assert [(b.kind, b.prefix or b.text) for b in reparsed] == [("ref", f"\\{prefix} "), ("paragraph", f"\\{prefix} text")] @@ -2058,14 +2058,14 @@ def test_a_model_built_reference_that_would_open_a_block_is_escaped(prefix): def _code_blocks(body: str) -> list[tuple[str, str]]: - document = parse_document(_FRONTMATTER + body) - assert parse_document(serialize_document(document)) == document + document = parse(_FRONTMATTER + body) + assert parse(serialize_document(document)) == document return [(b.kind, b.text) for b in document.sections[0].blocks] def test_directives_in_indented_code_are_literal(): body = "Text.\n\n {{ref: nope}} {#anchor} \"Fee\" {{def: fee}}\n {{bogus: x}}\n\nSee {{ref: anchor}}.\n" - result = validate_document(parse_document(_FRONTMATTER + body)) + result = validate(parse(_FRONTMATTER + body)) assert result.rules() == {"ref-broken"} assert [d.message for d in result.diagnostics] == ["Broken section reference: 'anchor'."] @@ -2096,14 +2096,14 @@ def test_where_indented_code_starts(body, kinds): def test_a_heading_indented_up_to_three_spaces_interrupts_a_paragraph(): - document = parse_document(_FRONTMATTER + "Text.\n ## Sub\n") + document = parse(_FRONTMATTER + "Text.\n ## Sub\n") assert [s.title for s in document.sections] == ["Terms", "Sub"] def test_a_document_can_open_with_indented_code(): - document = parse_document(" code\n\n# A\n") + document = parse(" code\n\n# A\n") assert [(b.kind, b.text) for b in document.preamble] == [("code", " code")] - assert parse_document(serialize_document(document)) == document + assert parse(serialize_document(document)) == document def test_indented_code_after_a_list_is_written_as_it_is(): @@ -2115,7 +2115,7 @@ def test_indented_code_after_a_list_is_written_as_it_is(): ]}]}) written = serialize_document(document) assert "- one\n\n x = `y`\n\n z" in written - assert parse_document(written) == document + assert parse(written) == document def test_every_indented_paragraph_after_a_list_is_the_items(): @@ -2124,7 +2124,7 @@ def test_every_indented_paragraph_after_a_list_is_the_items(): unindented paragraph ends the item (and the list).""" body = "1. Clause.\n\n Second.\n\n Pay {{placeholder: fee}}.\n\nThird.\n\n code {{ref: nope}}\n" assert [kind for kind, _text in _code_blocks(body)] == ["ordered_list", "paragraph", "code"] - result = validate_document(parse_document(_FRONTMATTER + body), final=True) + result = validate(parse(_FRONTMATTER + body), final=True) assert "placeholder-unfilled" in result.rules("error") and "ref-broken" not in result.rules() @@ -2134,8 +2134,8 @@ def test_an_indented_line_after_a_list_round_trips_whatever_it_begins_with(opene content: a block of the item's own.""" closer = f"\n {opener}" if opener in ("```", "~~~") else "" body = f"- a\n\n b\n\n {opener}{closer}\n\nd\n\n code\n" - document = parse_document(_FRONTMATTER + body) - assert parse_document(serialize_document(document)) == document + document = parse(_FRONTMATTER + body) + assert parse(serialize_document(document)) == document last = { "# foo": ("heading", "foo"), "
": ("html", "
"), "```": ("code", " ```\n ```"), "~~~": ("code", " ~~~\n ~~~"), "": ("html", " "), @@ -2163,9 +2163,9 @@ def test_a_table_with_an_indented_header_interrupts_a_paragraph(body, kinds): def test_an_indented_header_row_after_another_block_is_code(before): """Only an open paragraph can be interrupted by a table whose header row is indented; elsewhere that row is indented code (CommonMark).""" - document = parse_document(f"{_BARE}{before}\n | {{{{ref: nope}}}} |\n|---|\n") + document = parse(f"{_BARE}{before}\n | {{{{ref: nope}}}} |\n|---|\n") assert "table" not in [b.kind for _s, _i, b in document.iter_blocks()] - assert "ref-broken" not in validate_document(document).rules() + assert "ref-broken" not in validate(document).rules() def test_code_directly_after_a_list_is_written_so_it_stays_code(): @@ -2174,21 +2174,21 @@ def test_code_directly_after_a_list_is_written_so_it_stays_code(): (CommonMark), so the code after it is ordinary indented code, not the item's; only code with no paragraph between it and the list needs fencing to stay code rather than become the item's own text.""" - document = parse_document(_FRONTMATTER + "- a\n\n\n\n code {{ref: nope}}\n") - reparsed = parse_document(serialize_document(document)) + document = parse(_FRONTMATTER + "- a\n\n\n\n code {{ref: nope}}\n") + reparsed = parse(serialize_document(document)) assert [(b.kind, b.text) for b in reparsed.sections[0].blocks] == [ ("unordered_list", ""), ("paragraph", ""), ("code", " code {{ref: nope}}") ] - assert "ref-broken" not in validate_document(reparsed).rules() + assert "ref-broken" not in validate(reparsed).rules() document = document_from_dict({"sections": [{"title": "A", "blocks": [ {"kind": "unordered_list", "items": ["a"]}, {"kind": "code", "text": " x"}, ]}]}) - blocks = parse_document(serialize_document(document)).sections[0].blocks + blocks = parse(serialize_document(document)).sections[0].blocks assert [(b.kind, b.text) for b in blocks] == [("unordered_list", ""), ("code", " x")] document = document_from_dict({"sections": [{"title": "A", "blocks": [ {"kind": "unordered_list", "items": ["a"]}, {"kind": "paragraph", "text": "# foo"}, {"kind": "code", "text": " x"}, ]}]}) - blocks = parse_document(serialize_document(document)).sections[0].blocks + blocks = parse(serialize_document(document)).sections[0].blocks assert [(b.kind, b.text) for b in blocks] == [("unordered_list", ""), ("paragraph", "\\# foo"), ("code", " x")] @@ -2197,7 +2197,7 @@ def test_text_after_a_list_that_would_open_a_block_is_escaped(): {"kind": "unordered_list", "items": ["a"]}, {"kind": "paragraph", "text": "> q"}, {"kind": "paragraph", "text": "# foo"}, ]}]}) - blocks = parse_document(serialize_document(document)).sections[0].blocks + blocks = parse(serialize_document(document)).sections[0].blocks assert (blocks[-1].kind, blocks[-1].text) == ("paragraph", "\\# foo") @@ -2217,7 +2217,7 @@ def test_text_after_a_list_that_would_open_a_block_is_escaped(): ) def test_table_rows_are_indented_at_most_three_columns_and_have_cells(body, kinds): assert [kind for kind, _text in _code_blocks(body)] == kinds - assert "ref-broken" not in validate_document(parse_document(_FRONTMATTER + body)).rules() + assert "ref-broken" not in validate(parse(_FRONTMATTER + body)).rules() @pytest.mark.parametrize("tag", ["
", "